sync: auto-sync from GURU-5070 at 2026-06-18 12:49:38

Author: Mike Swanson Machine: GURU-5070 Timestamp: 2026-06-18 12:49:38
dataforth/testdatadb: 8B/5B/SCM render verify results + convergence plan
2026-06-18 12:51:08 -07:00 · 2026-06-18 12:50:44 -07:00 · 2026-06-18 12:50:44 -07:00 · 2026-06-18 12:50:44 -07:00 · 2026-06-18 12:50:44 -07:00 · 2026-06-18 12:50:43 -07:00
14 changed files with 12095 additions and 180 deletions
--- a/.claude/memory/MEMORY.md
+++ b/.claude/memory/MEMORY.md
@@ -32,6 +32,9 @@
 - [AAD Connect msDS-KeyCredentialLink writeback](reference_aadconnect_keycredlink_writeback.md) — "completed-export-errors" + 8344 INSUFF_ACCESS_RIGHTS on a protected admin account = WHfB key writeback blocked by AdminSDHolder. Diagnose with csexport /f:x; fix with dsacls WP;msDS-KeyCredentialLink on AdminSDHolder + SDProp.
 - [UniFi Site Manager cloud API](reference_unifi_site_manager_api.md) — `api.ui.com` + `X-API-KEY` (vault `services/unifi-site-manager`) = remote access to the WHOLE ACG UniFi fleet (~36 consoles) outside UOS. Tier1 `/v1/hosts|sites|devices|isp-metrics` = inventory+health+WAN. Tier2 CONNECTOR `/v1/connector/consoles/{id}/proxy/network/api/s/default/stat/{device,sta}` = **full UOS parity** (per-radio cu_total airtime + per-client RSSI) for ANY console, remote. Backend `unifi-wifi/scripts/gw-sitemanager.sh` (`fleet|devices|sites|isp|net`). Standalone UDM WAN SSH usually firewalled; per-console SSH pw at `clients/<slug>/udm-ssh`.
 - [reference_sqlx_migrations_immutable](reference_sqlx_migrations_immutable.md) -- NEVER edit an already-applied sqlx migration file — even a comment. sqlx::migrate! checksums each file at compile time and validates against _sqlx_migrations at startup; a changed checksum crash-loops the server with "migration N was previously applied but has been modified". Code review MUST flag any edit to an applied migration.
+- [AD2 SSH MTU blackhole](ad2-ssh-mtu-blackhole.md) — AD2 SSH "lockouts"/mid-session read-errors over the Dataforth OpenVPN were a PMTU blackhole (tunnel PMTU ~1424 vs adapter MTU 1500), NOT a ban/account-lockout/flaky tunnel. Fix: pin the OpenVPN adapter MTU to 1400 (done on GURU-5070 via its SYSTEM RMM agent); permanent = `mssfix 1360` on the OpenVPN server. Diagnose over RMM, not SSH.
+- [DSCA33/45 resolved via Hoffman](project_dsca33_45_resolved_via_hoffman.md) — The "lost" DSCA33/45 spec files are recoverable from the Hoffman API (original certs survived the wipe); do NOT ask John. 56/58 models mined into projects/dataforth-dos/dsca33-45-templates.json; only DSCA33-1948 + DSCA45-1746 (24 units) lack an original. AD2 handoff: DSCA33-45-HOFFMAN-RECOVERY-2026-06-18.md.
+- [AD2 comms via sync only](ad2-comms-via-sync-only.md) — The AD2 Dataforth-box Claude session is coord-API-isolated (Gitea only); coord msg/lock/todo never reach it. Coordinate with AD2 ONLY via git /sync (committed docs + ## Note blocks).

 ## Users
 - [Howard Enos](user_howard.md) — Mike's brother, technician, full access. Machines: ACG-TECH03L, Howard-Home (authoritative in users.json).
--- a/.claude/memory/ad2-comms-via-sync-only.md
+++ b/.claude/memory/ad2-comms-via-sync-only.md
@@ -0,0 +1,21 @@
+---
+name: ad2-comms-via-sync-only
+description: The AD2 (Dataforth) Claude session is coord-API-isolated — reach it ONLY via git /sync (committed notes/docs), never coord messages/locks
+metadata:
+  type: feedback
+---
+
+The AD2 Dataforth-box Claude session is **network-isolated from the ACG coord API** (172.16.3.30 —
+the Dataforth network can't reach it); it only has Gitea/git access. So coord-API **messages, locks,
+and todos NEVER reach AD2**. ALL inter-session coordination with AD2 must go through git **`/sync`**:
+committed handoff docs and `## Note for <user>` blocks in synced session logs, which AD2 reads when
+it pulls. A coord lock on an AD2-only file (e.g. `datasheet-exact.js`) is also meaningless — only the
+AD2 session edits that box.
+
+**Why:** burned a round of `coord msg send AD2` + lock that were silent no-ops (Mike: "You can't
+coord with AD2 — all comms needs to be via sync").
+
+**How to apply:** to hand work to or coordinate with the AD2 session, write it into a committed doc
+(e.g. `projects/dataforth-dos/*HANDOFF*.md`) and/or a `## Note for <user>` block in a session log,
+then `/sync`. Do NOT use the coord skill for AD2. (Coord API is still fine for non-isolated ACG
+machines.) [[prefer-ssh-over-rmm]]
--- a/.claude/memory/ad2-ssh-mtu-blackhole.md
+++ b/.claude/memory/ad2-ssh-mtu-blackhole.md
@@ -0,0 +1,40 @@
+---
+name: ad2-ssh-mtu-blackhole
+description: AD2 SSH "lockouts"/mid-session timeouts over the Dataforth OpenVPN were an MTU/PMTU blackhole, not a ban/account-lockout/flaky tunnel; fix = pin the tunnel adapter MTU to 1400
+metadata:
+  type: project
+---
+
+AD2 (Dataforth, `192.168.0.6`) SSH from the fleet over OpenVPN (client subnet `192.168.6.x`)
+intermittently looked "locked out": sessions **authenticated fine**, then died mid-session with
+`Read error from remote host 192.168.6.2 ... Unknown error [postauth]` and
+`ssh_dispatch_run_fatal: Connection from authenticating user sysadmin ... Connection timed out [preauth]`.
+Small/interactive commands often worked; bulk reads + `scp` stalled.
+
+**Root cause (diagnosed 2026-06-18 via RMM — SSH itself was the failing channel, so don't diagnose it over SSH):**
+- NOT account lockout — Windows lockout threshold is 5/30min but **zero 4740 events**; `sysadmin` never locked.
+- NOT an IP ban — **no IPBan/wail2ban/RdpGuard**, **0 inbound firewall block rules**.
+- NOT auth — **every** `Accepted publickey for sysadmin` succeeded.
+- NOT load — AD2 was CPU ~11%, 11.7 GB RAM free.
+- It was a **PMTU blackhole.** OpenVPN tunnel path MTU is **~1424** (DF ping: wire 1424 passes,
+  1428 drops). But GURU-5070's OpenVPN adapter (`Local Area Connection`, ifIndex 12, IP
+  `192.168.6.2`) was set to **MTU 1500** → TCP negotiated MSS 1460 → full-size bulk/scp segments
+  exceeded the tunnel and were **silently dropped (DF set)**, while sub-MTU interactive packets
+  passed. That is why it presented as random "lockouts" that got worse with bulk transfer.
+
+**Fix applied (2026-06-18):** `Set-NetIPInterface -InterfaceIndex 12 -AddressFamily IPv4 -NlMtuBytes 1400`
+run via **GURU-5070's own RMM agent** (`819df0c8...`, runs as `nt authority\system` = elevated; the
+elevated lever on the local box when you can't self-elevate from the Claude shell). Validated: a
+**1.41 MB single-session SSH transfer to AD2 completed in 9s, no read error** (previously blackholed).
+`~/.ssh/config` `ad2` block annotated + tightened keepalives (`ServerAliveInterval 15`,
+`ServerAliveCountMax 4`, `ConnectTimeout 20`).
+
+**Durability / permanent fix:** `Set-NetIPInterface` is registry-persistent, but **OpenVPN Connect may
+reset the adapter MTU to 1500 on reconnect** — re-apply if SSH bulk transfers start stalling again
+(check `Get-NetIPInterface -InterfaceIndex 12`). The real permanent fix is **server-side on the
+Dataforth OpenVPN server: `mssfix 1360` (or `push "tun-mtu 1400"`)** so every fleet client clamps
+automatically — `192.168.6.4` showed the identical symptom, so this is fleet-wide, not 5070-only.
+
+Corrects the earlier wrong attribution ("flaky VPN tunnel" / "my rapid scp+ssh bursts triggering a
+ban") — the tunnel is up and stable for small packets; only over-MSS segments were dropped. See
+[[prefer-ssh-over-rmm]] (RMM-as-fallback guidance still holds; the *reason* was MTU, not a flaky VPN).
--- a/.claude/memory/project_dsca33_45_resolved_via_hoffman.md
+++ b/.claude/memory/project_dsca33_45_resolved_via_hoffman.md
@@ -0,0 +1,27 @@
+---
+name: project_dsca33_45_resolved_via_hoffman
+description: DSCA33/45 "lost spec files" are recoverable from the Hoffman API (original certs survived the wipe) — do NOT request spec files from John; mine templates from Hoffman instead
+metadata:
+  type: project
+---
+
+The DSCA33/DSCA45 main spec files lost in Dataforth's cryptolocker wipe (which blocked rendering
+~8,763 certs and prompted "ask John for the spec files") are **recoverable** — the original software
+published correct DSCA33/45 certs to the **Hoffman API** before the wipe, and our null-skipping
+pipeline never overwrote them. **Do not ask John for spec files.** Supersedes the FIX2-5 handoff's
+TODO 2 and the `ad2`-branch memory `project_dsca33_45_spec_gap` (which says "blocked, need John").
+
+Mined **56 of 58 models** straight from Hoffman into `projects/dataforth-dos/dsca33-45-templates.json`
+(per model: `accOut`, verbatim 2-line `accHeader`, Final-Test `rows` of name+spec, and a known-good
+`_srcSerial`). Only 2 niche models have no original anywhere: **DSCA33-1948 (16u)**, **DSCA45-1746 (8u)**.
+Coverage: ~7,157 units already correct + live on Hoffman (no action); ~1,580 not-yet-uploaded units
+need rendering from the mined templates + AD2's already-derived slotMaps.
+
+AD2 handoff + the critical gate: `projects/dataforth-dos/DSCA33-45-HOFFMAN-RECOVERY-2026-06-18.md`.
+**Critical:** validate each model's render byte-for-byte against its Hoffman original BEFORE enabling
+live DSCA33/45 rendering — once the renderer returns non-null, the pipeline stops skipping these and
+will re-push/UPDATE the 7,157 good originals on the next cycle (safe only if the render matches).
+
+Hoffman read API: `GET https://www.dataforth.com/api/v1/TestReportDataFiles/{serial}` (returns
+`{SerialNumber,Content,CreatedAtUtc,UpdatedAtUtc}`); creds vault `clients/dataforth/hoffman-product-api`.
+Miner: `projects/dataforth-dos/tools/mine-hoffman-dsca.py`. AD2 access notes: [[ad2-ssh-mtu-blackhole]].
--- a/clients/darrell-delphen/session-logs/2026-06/2026-06-18-mike-email-link-sni-block.md
+++ b/clients/darrell-delphen/session-logs/2026-06/2026-06-18-mike-email-link-sni-block.md
@@ -0,0 +1,119 @@
+# 2026-06-18 — Darrell Delphen — Outlook email links failing (ISP SNI block)
+
+## User
+- **User:** Mike Swanson (mike)
+- **Machine:** GURU-5070
+- **Role:** admin
+
+## Session Summary
+
+Darrell Delphen reported that links in Outlook email would not open on one workstation
+(DDDOffice072023), failing with "can't reach this page" / `ERR_CONNECTION_ABORTED` against a
+`url.emailprotection.link` URL, while links opened from Gmail worked fine. The failing host is
+Intermedia's Email Protection "Safe Link" rewriter — every link in Intermedia-protected mail is
+rewritten to `https://url.emailprotection.link/...`, so all Outlook-delivered links routed through it.
+
+Diagnosis was done entirely over GuruRMM against agent `000ed57d-fd05-4001-871c-244f43155c16`. DNS
+resolved correctly and TCP 443 connected, but the TLS handshake died with SChannel `0x80090326`
+(SEC_E_ILLEGAL_MESSAGE — "message received was unexpected or badly formatted"). The endpoint's TLS
+stack was clean: FIPS off, no SCHANNEL protocol/cipher overrides, no cipher-suite GPO, only Windows
+Defender (no third-party AV/proxy/VPN/LSP/WFP callout). The same node `199.193.205.140` handshook
+successfully with the real SNI from GURU-5070 on a different network, proving the origin was healthy
+and the interference was on the endpoint's path. A blast-radius sweep showed only
+`url.emailprotection.link` failed while `google.com`, `microsoft.com`, `outlook.office365.com`,
+`cloudflare.com`, `badssl.com`, and even `login.serverdata.net` (same Intermedia infra) all succeeded;
+MTU was fine. An SNI-varied handshake to the same IP isolated it conclusively: `example.com` and
+`login.serverdata.net` SNIs succeeded 12/12 while `url.emailprotection.link` failed 12/12, across
+interleaved source ports — deterministic, SNI-keyed, not flow-hash/LAG. Root cause: a network device
+on the path performing SNI-based content inspection that corrupted the handshake for that one hostname.
+
+The gateway turned out to be an ISP-provided Extreme **EXOS** device the client had no login for. The
+fix path was therefore ISP escalation. An escalation packet was drafted, and Cloudflare WARP was
+installed on the workstation as an interim workaround — tunneling past the SNI block. With WARP
+connected, egress moved to Cloudflare (104.28.152.216) and the real-SNI handshake succeeded (TLS 1.2,
+HTTP 200). The ISP then disabled a "NetIQ" web/URL-filtering feature on the gateway, which cleared the
+block at the source. After WARP was disconnected the native ISP path was verified working (5/5
+handshakes + HTTP 200, egress back to 167.89.210.225), so WARP was uninstalled and the machine
+returned to normal. Work was documented and billed on Syncro ticket #32437 — private technical note,
+customer-facing/emailed summary, and 1.0h remote labor ($150.00, invoice #1650728058).
+
+## Key Decisions
+
+- Diagnosed exclusively via repeated GuruRMM `SslStream`/`Test-NetConnection` probes rather than
+  asking the client to run tools — faster and reproducible.
+- Used an SNI-varied handshake to the *same fixed IP* as the decisive test. It separated
+  destination/server problems from path interference and proved the block was keyed on the SNI string.
+- Ran a 12x repeatability test per SNI to rule out a faulty LAG/ECMP member (intermittent, 5-tuple
+  keyed) vs deliberate content matching (deterministic). Result was 0/12 vs 12/12 — deterministic.
+- Chose Cloudflare WARP as the interim workaround because the block is SNI-based; any tunnel that
+  encrypts egress past the EXOS hides the SNI. WARP is the lightest deploy.
+- Installed/connected WARP in stages (install, then connect, then verify) so each step could be
+  confirmed before flipping the tunnel, given the agent bounces on network-stack changes.
+- Emailed the customer a plain-language summary (do_not_email:false) and kept the technical detail in
+  a hidden note.
+
+## Problems Encountered
+
+- **Shell state not persisting between Bash calls** — `$TOKEN`/`$RMM` from `rmm-auth.sh` were gone on
+  the next call (first dispatch produced no output). Fixed by `eval "$(rmm-auth.sh)"` inside every Bash
+  invocation.
+- **PowerShell single-quotes collided with bash single-quoted `SCRIPT='...'`** — embedded
+  `'C:\Program Files\...'` terminated the bash string (`FilesCloudflareCloudflare: command not found`).
+  Fixed by defining the script via a quoted heredoc `SCRIPT=$(cat <<'PS' ... PS)`.
+- **WARP install/connect bounced the RMM agent** — commands returned `interrupted` ("Agent restarted
+  during execution") because installing/connecting WARP resets the network stack/WFP. The agent
+  auto-reconnected (through WARP after connect); verified state with follow-up commands.
+- **First WARP uninstall found nothing** — the product registers as **"Cloudflare One Client"**, not
+  "Cloudflare WARP", so the DisplayName filter missed it. Found the GUID
+  `{9E49837E-2971-413F-9587-119FA819E572}` and removed via `msiexec /x`.
+
+## Configuration Changes
+
+- **Endpoint DDDOffice072023:** Cloudflare WARP (Cloudflare One Client 2026.4.1390.0) installed,
+  registered, connected, then later disconnected and **fully uninstalled**. Net change to the machine: none.
+- **ISP gateway (not us):** ISP disabled the "NetIQ" web/URL-filtering feature on the Extreme EXOS device.
+- No changes to the ClaudeTools repo code.
+
+## Credentials & Secrets
+
+None discovered, created, or rotated this session.
+
+## Infrastructure & Servers
+
+- **Endpoint:** DDDOffice072023 — Windows, GuruRMM agent `000ed57d-fd05-4001-871c-244f43155c16`
+  (v0.6.66), RMM client "AZ Computer Guru" / site "Discovery test site". LAN gateway 192.168.1.1,
+  ISP egress 167.89.210.225, WARP egress (while active) 104.28.152.216.
+- **ISP gateway:** Extreme Networks **EXOS** device, ISP-provided/managed (client has no login). Was
+  running a "NetIQ" URL-filtering feature doing SNI inspection.
+- **Blocked destination:** `url.emailprotection.link` → CNAME `urlrs.gslb.serverdata.net` → A
+  199.193.205.140 (Intermedia Email Protection link-rewriter). GSLB pool also advertises
+  199.193.200.65 / 64.78.20.65 / 162.244.196.65, all TCP-dead from any network (not an ISP block).
+- **GuruRMM API:** http://172.16.3.30:3001 (JWT via `rmm-auth.sh`).
+
+## Commands & Outputs
+
+- Decisive SNI test (same IP, varied SNI), via RMM PowerShell:
+  - SNI `url.emailprotection.link` → `0x80090326` SEC_E_ILLEGAL_MESSAGE (0/12 ok)
+  - SNI `example.com` / `login.serverdata.net` → TLS 1.2 AES256 (12/12 ok)
+- Off-network control (GURU-5070) to 199.193.205.140 with real SNI → OK TLS 1.2.
+- Post-ISP-fix native verify: real-SNI handshake 5/5 OK + `Invoke-WebRequest` HEAD → HTTP 200,
+  egress 167.89.210.225.
+- WARP removal: `msiexec /x {9E49837E-2971-413F-9587-119FA819E572} /qn /norestart` → exit 0;
+  warp-svc/warp-cli/install-dir/uninstall-entry all gone.
+
+## Pending / Incomplete Tasks
+
+- None functional. Block is resolved at the ISP. If the issue recurs, suspect the EXOS "NetIQ"
+  feature being re-enabled — re-run the SNI-varied handshake test to confirm.
+- Optional: if WARP is ever rolled to more machines as a workaround, harden it (force auto-connect,
+  lock client) and note egress moves to Cloudflare (breaks office-static-IP allowlisting).
+
+## Reference Information
+
+- **Syncro ticket:** #32437 (id 112766479) — https://computerguru.syncromsp.com/tickets/112766479
+  - Private note id 419714810; public/emailed summary id 419714813
+  - Line item id 42925426 (Labor - Remote Business 1.0h @ $150)
+  - Invoice #1650728058, total $150.00; status Invoiced
+- **Customer:** Darrell Delphen (Syncro customer_id 35996725), no prepaid block.
+- **SChannel error:** 0x80090326 = SEC_E_ILLEGAL_MESSAGE (handshake message malformed/unexpected) —
+  signature of in-path TLS/SNI tampering when paired with same-IP success off-network.
--- a/errorlog.md
+++ b/errorlog.md
@@ -21,6 +21,16 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure ·

 2026-06-18 | Howard-Home | rmm | [friction] agent returns exit -1 'Failed to execute command' on a ~7KB multi-line powershell body sent as one command; split into <2KB section scripts and each ran fine [ctx: host=DESKTOP-TRCIEJA agent=0.6.66]

+2026-06-18 | GURU-5070 | coord/ad2-comms | [correction] tried to coordinate with the AD2 session via coord API msg+lock; AD2 is network-isolated (Gitea only, no coord API) so those were no-ops. ALL inter-session comms with AD2 must go via git /sync (committed notes/docs).
+
+2026-06-18 | GURU-5070 | syncro | comment POST piped straight to jq failed with 'jq: parse error: Invalid numeric literal at line 1 col 10' and left it AMBIGUOUS whether the note posted (GET-verify showed it had NOT); per no-retry rule had to GET first, then re-post. Robust pattern that worked: jq -n payload to a file, POST with --data-binary @file, capture response to a file, then GET-verify by subject. Skill's curl|jq comment pattern should adopt this. [ctx: ticket=32441 skill=syncro pattern=curl-pipe-jq]
+
+2026-06-18 | GURU-5070 | post-bot-alert | Discord POST failed (non-200/unreachable) [ctx: channel=#bot-alerts http=400 resp={"message": "The request body contains invalid JSON.", "code": 50109}]
+
+2026-06-18 | GURU-5070 | ssh/ad2 | [correction] attributed AD2 SSH timeouts to a flaky VPN tunnel + my rapid scp/ssh bursts; real cause = OpenVPN adapter MTU 1500 vs tunnel PMTU ~1424 -> TCP MSS blackhole that drops bulk/scp segments (DF set) while small cmds pass. Fix: tunnel adapter MTU 1400 [ctx: ref=feedback_prefer_ssh_over_rmm]
+
+2026-06-18 | GURU-5070 | bash/env | [friction] /tmp curl-write then Windows-python read mismatch; wrote .claude/tmp + absolute path fixed it [ctx: ref=feedback_tmp_path_windows]
+
 2026-06-18 | Howard-Home | pfsense-ssh/logs | [friction] used clog on pfSense 25.07 logs (now plain-text ASCII) -> empty output -> wrongly concluded DHCP log was empty / dhcpd not serving; cost a hypothesis. Read pfSense 25.07 logs with tail/grep/cat directly, NOT clog [ctx: ref=reference_pfsense_25_07_ops client=cascades-tucson]

 2026-06-17 | GURU-5070 | mailbox/365-mail | [correction] claimed in a prior session that /mailbox skill + memories were repointed off the deleted fabb3421 to the 365-mail suite, but mailbox.md still hardwired fabb3421 (token 401 AADSTS700016). Correct app is the dedicated ComputerGuru Mailbox app 1873b1b0 via get-token.sh 'mailbox' tier (cert auth); repointed mailbox.md + feedback_365_remediation_tool.md 2026-06-17. Lesson: verify the edit actually landed before reporting it done.
--- a/projects/dataforth-dos/8B5BSCM-RENDER-VERIFY-2026-06-18.md
+++ b/projects/dataforth-dos/8B5BSCM-RENDER-VERIFY-2026-06-18.md
@@ -0,0 +1,42 @@
+# 8B/5B/SCM Render Fix — Diagnosis + Stage/Verify Results (2026-06-18)
+
+Driving the ~5,148 unpublished 8B/5B/SCM PASS records (render-coverage gap, same class as DSCA).
+
+## Root cause (validated)
+1. **parseRawData bug (general):** a PASS/FAIL line is wrongly consumed as the step-response line
+   for non-DSCA families that omit the `"0","0",v` line (8B45/8B49/5B39/SCM5B33...) -> drops the
+   first Final-Test group -> measurement-count mismatch -> null. Fix: change the step-skip guard to
+   `if (!((family === 'DSCA' || skipStepIfStatus) && looksLikeStatus))` and pass `skipStepIfStatus`
+   only for templated models (so non-templated models are byte-unchanged).
+2. **No per-model Final-Test template:** one hardcoded `DATA_LINES['8B']` (RTD-shaped) -> wrong
+   param names/specs for non-RTD models (8B45 is frequency-input == DSCA45). Needs per-model
+   templates, same as DSCA. The "published" siblings render null too (legacy content on Hoffman;
+   `api_uploaded_at` was back-populated from Hoffman inventory, NOT from our render).
+
+## Artifacts
+- `8b5bscm-templates.json` — 136 models mined from Hoffman originals (accOut, accHeader, rows, _srcSerial).
+- Verify harness pattern: patch a TEMP copy of datasheet-exact.js (parse-fix + template-gated
+  Final-Test, footer skip), render each model's published `_srcSerial`, content-normalized compare
+  to its Hoffman original. (Never touches the live renderer.)
+
+## Verify results (template-gated Final-Test only; NO slotmaps/precision/accuracy-label work yet)
+Content-only (cosmetic header/spacing deferred, per the DSCA byte-fidelity decision):
+- 15 models content-perfect (>=0.99) — publishable as-is.
+- 17 within precision-tuning distance (0.93-0.97) — single value-row rounding diffs.
+- 27 @ 0.85-0.93, 7 < 0.85.
+- 70 NULL — render-count mismatch -> need per-model slotmaps.
+- By family: 8B avg 0.97, SCM5B 0.93 (strong); 8B38 0.78; 7B ~0.88 (separate parse path).
+
+## Remaining work = the SAME machinery AD2 built for DSCA
+1. Per-model **slotmaps** for the 70 nulls (AD2 `_derive_slotmaps.js`) — derive by matching rendered
+   measured values to the Hoffman original's displayed values.
+2. **QB single-precision rounding** (AD2 `Math.fround`) — closes the 0.93-0.97 precision diffs.
+3. **Frequency/AAC accuracy-block labels + stimulus formatting** — the still-open DSCA45/33 piece
+   (8B45/8B49/5B45 are frequency-input; SCM5B33 is AAC). Solving once covers BOTH 8B/5B AND DSCA45/33.
+4. 8B38 (0.78) + 7B family (separate SCM7B path) — family-specific.
+
+## Recommendation
+Converge with AD2's mature DSCA machinery rather than re-implement in parallel in the shared
+`datasheet-exact.js`: feed the mined `8b5bscm-templates.json` into AD2's template/slotmap/rounding
+path and finish the frequency/AAC accuracy renderer once for DSCA + 8B/5B/SCM together.
+The 15 content-clean models can be published immediately if a partial win is wanted.
--- a/projects/dataforth-dos/8b5bscm-templates.json
+++ b/projects/dataforth-dos/8b5bscm-templates.json
--- a/projects/dataforth-dos/DSCA33-45-HOFFMAN-RECOVERY-2026-06-18.md
+++ b/projects/dataforth-dos/DSCA33-45-HOFFMAN-RECOVERY-2026-06-18.md
@@ -0,0 +1,106 @@
+# DSCA33 / DSCA45 — Recover the "lost" specs from Hoffman (Handoff to AD2)
+
+**For:** the Claude session on AD2 (`C:\Shares\testdatadb`). **Ref:** Syncro #32441.
+**Supersedes** the FIX2-5 handoff's *TODO 2 (request DSCA33/45 main spec files from John)* —
+**we do not need John.** The original specs are recoverable from the Hoffman API.
+
+---
+
+## The finding (why this changes the plan)
+
+The DSCA33/DSCA45 "main spec" records (DSCMAIN/DSCOUT with SENTYPE/MAXIN/input-type) were
+lost in the cryptolocker wipe, so `render-datasheet.js` bails (null render) and the pipeline
+**skips** those models. BUT the **original software published correct DSCA33/45 certs to the
+Hoffman API before the wipe**, and our broken renderer never overwrote them (it skips null
+renders). They are still there, pristine.
+
+Verified from GURU-5070 against the public API
+(`GET https://www.dataforth.com/api/v1/TestReportDataFiles/{serial}`, OAuth client-creds,
+vaulted `clients/dataforth/hoffman-product-api`):
+
+| Family | Models | Mineable from Hoffman | Units already correct+live on Hoffman | No original anywhere |
+|---|---|---|---|---|
+| DSCA33 | 35 | **34** | 2,633 / 3,397 | **DSCA33-1948 (16 units)** |
+| DSCA45 | 23 | **22** | 4,524 / 5,413 | **DSCA45-1746 (8 units)** |
+
+So of the ~8,763 "blocked" certs: **~7,157 are already correct and live on the public site**
+(no action needed), **~1,580 not-yet-uploaded units** just need rendering, and only
+**24 units across 2 niche models** have no original to recover.
+
+---
+
+## The artifact (already built — use it, don't re-derive)
+
+`projects/dataforth-dos/dsca33-45-templates.json` — **56 models**, mined from the Hoffman
+originals with the same `===`-rule column-span extractor as STAGE 1. Schema is a **superset of
+`dsca-templates.json`**:
+
+```json
+"DSCA45-05E": {
+  "accOut": "Output (mA)",
+  "accHeader": [
+    "        Frequency    Calculated      Measured",
+    "           (Hz)      Output (mA)    Output (mA)*    Error (%)     Status"
+  ],
+  "rows": [ {"name":"Supply Current","spec":"<  105 mA"}, ... ],
+  "_srcSerial": "176326-2"
+}
+```
+
+- `rows` = the Final-Test `Parameter | Specification` list (incl. the spec-less rows like
+  `240 VAC Withstand`, `Hi-Pot`, and section sub-heads `Zero-Crossing Input` / `TTL Input` —
+  **kept**, same skip-rule reconciliation as STAGE 2).
+- `accHeader` = the **verbatim 2-line accuracy header** from the original. Use it — DSCA33/45
+  introduce header tokens the 92-model set never had (see flags) and the frequency-input layout.
+- `_srcSerial` = a known already-uploaded serial for that model → your validation oracle.
+
+Spot-checked DSCA33-07C and DSCA45-05E row-for-row against their live Hoffman originals: exact.
+
+Regenerate if needed: `python projects/dataforth-dos/tools/mine-hoffman-dsca.py <map.json> <out.json>`.
+
+---
+
+## Flags (DSCA33/45 differ from the 92 you already did)
+
+1. **New accOut tokens:** `Output (VDC)` and `Output (mADC)` (DSCA33 current/voltage DC outputs),
+   not just `Output (V)`/`Output (mA)`. Your accuracy-block must emit the verbatim token.
+2. **Model-specific accuracy input label:** DSCA33 uses `Vin (mVAC)` / `Vin (VAC)` / `Iin (AAC)` /
+   `Iin (mAAC)`. **Use the `accHeader` lines** rather than synthesizing from a (missing) spec field.
+3. **DSCA45 is frequency-input:** two-line super-header `Frequency` / `(Hz)` and a frequency
+   sweep in the accuracy block — structurally unlike the voltage/current-input models. Confirm the
+   accuracy renderer reproduces it (the `accHeader` gives you the exact text).
+4. Your **DSCA33/45 slotMaps are already derived** and come from the same original layout as these
+   rows, so order should align — verify during validation.
+
+---
+
+## Plan
+
+1. **Backup first** (fresh `pg_dump` + VSS) per FIX2-5 discipline. Save-state `datasheet-exact.js`.
+2. **Load** `dsca33-45-templates.json` for `family in (DSCA33, DSCA45)`, same wiring as STAGE 2
+   (template `rows` drive names+specs; map raw_data STATUS groups positionally via the slotMaps;
+   QB `Math.fround` rounding; data-driven loadNote). For the accuracy block, drive the header from
+   `accHeader` / `accOut`.
+3. **VALIDATE against Hoffman (the gate — stronger than STAGE 3):** for each model, render its
+   `_srcSerial` (an already-uploaded unit) and **content-normalized byte-compare against
+   `GET /api/v1/TestReportDataFiles/{_srcSerial}`**. Require a clean match **per model** before that
+   model is allowed to render/publish.
+   > **CRITICAL — do not enable live DSCA33/45 rendering until each model passes.** The moment the
+   > renderer returns non-null for DSCA33/45, the pipeline stops skipping them and will **re-push and
+   > UPDATE the ~7,157 already-correct originals** on the next cycle. That is only safe if the render
+   > byte-matches the original — which the per-model gate proves. A mismatched model would overwrite
+   > good customer certs. Gate hard.
+4. **Publish the gap:** for validated models, render + `uploadBySerialNumbers` the **not-yet-uploaded**
+   units (~1,580; `api_uploaded_at IS NULL`). Already-uploaded units return `Unchanged` (idempotent).
+5. **Leave blocked (24 units):** `DSCA33-1948` (16), `DSCA45-1746` (8) — no Hoffman original. Low
+   priority; only these would ever need John, and they look like one-off custom part numbers.
+
+Commit to `ad2`; update #32441 (hidden notes). The remote operator sees it on sync.
+
+---
+
+## Reference
+- Templates: `projects/dataforth-dos/dsca33-45-templates.json` (56 models)
+- Miner: `projects/dataforth-dos/tools/mine-hoffman-dsca.py`
+- Hoffman API creds: vault `clients/dataforth/hoffman-product-api` (read = `GET .../TestReportDataFiles/{serial}`)
+- Memory: `project_dsca33_45_spec_gap` (updated — resolved via Hoffman, not John)
--- a/projects/dataforth-dos/dsca33-45-templates.json
+++ b/projects/dataforth-dos/dsca33-45-templates.json
--- a/projects/dataforth-dos/testdatadb-fix/public/index.redesign.html
+++ b/projects/dataforth-dos/testdatadb-fix/public/index.redesign.html
@@ -12,7 +12,7 @@
    --hover:#f1f5f9; --sel:#e0e7ff; --desk:#e7ecf1;
    --mono:ui-monospace,"SFMono-Regular",Consolas,"Liberation Mono",monospace;
    --sans:Inter,system-ui,-apple-system,"Segoe UI",Roboto,sans-serif;
-    --r:6px; --insp:480px;
+    --r:6px; --insp:500px;
  }
  *{box-sizing:border-box}
  html,body{height:100%;margin:0}
@@ -112,6 +112,10 @@
  .pill{display:inline-block;font-size:10.5px;font-weight:700;padding:1px 8px;border-radius:4px;font-family:var(--mono)}
  .pill.PASS{background:var(--pass-bg);color:var(--pass-ink)}
  .pill.FAIL{background:var(--fail-bg);color:var(--fail-ink)}
+  .tag{display:inline-flex;align-items:center;gap:5px;font-size:11px;font-weight:600;padding:2px 8px;border-radius:20px;font-family:var(--sans)}
+  .tag::before{content:"";width:7px;height:7px;border-radius:50%}
+  .tag.pub{background:var(--pass-bg);color:var(--pass-ink)} .tag.pub::before{background:var(--pass-ink)}
+  .tag.unpub{background:#fef3c7;color:#92400e} .tag.unpub::before{background:#d97706;background:none;box-shadow:inset 0 0 0 1.5px #d97706}
  .web{font-size:13px}
  .pager{display:flex;align-items:center;gap:10px;padding:7px 12px;border-top:1px solid var(--border);font-size:12px;color:var(--ink-2)}
  .pager button{font-size:12px;height:28px;padding:0 11px;border:1px solid var(--border-strong);border-radius:var(--r);background:#fff;cursor:pointer;color:var(--ink)}
@@ -200,7 +204,7 @@
    <div class="selbar" id="selbar">
      <span id="selcount">0 selected</span>
      <button id="copySel">Copy serials</button>
-      <button id="repushSel" disabled title="Re-publish to the website — needs an API endpoint">Re-push ▴</button>
+      <button id="repushSel" title="Publish / re-publish the selected serials to the public website">Re-push ▴</button>
      <span class="sp" style="flex:1"></span>
      <button id="selclear" style="border:0;background:none">clear</button>
    </div>
@@ -240,7 +244,7 @@
 const API='';
 const $=id=>document.getElementById(id);
 const state={q:'',serial:'',model:'',result:'',station:'',logtype:'',from:'',to:'',size:50,page:0,total:0,
-  selected:null,rows:[],sort:'',dir:'desc',checks:new Set(),force:''};
+  selected:null,rows:[],sort:'',dir:'desc',checks:new Set(),force:'',webStatus:''};
 let timer=null, certTimer=null;
 const esc=s=>String(s==null?'':s).replace(/[&<>"]/g,c=>({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;'}[c]));
 const fmtDate=d=>d?String(d).slice(0,10):'';
@@ -265,7 +269,8 @@ $('mode').onclick=()=>{ const seq=['auto','serial','model','text']; state.force=
 function params(forExport){
  const p=new URLSearchParams();
  for(const k of ['q','serial','model','result','station','logtype','from','to']) if(state[k]) p.set(k,state[k]);
-  if(state.sort){ p.set('sort',state.sort); p.set('dir',state.dir); }   // needs API; harmless if ignored
+  if(state.sort){ p.set('sort',state.sort); p.set('dir',state.dir); }   // honored by /api/search (whitelisted)
+  if(state.webStatus) p.set('web_status',state.webStatus);              // on=published, off=not yet published
  if(!forExport){ p.set('limit',state.size); p.set('offset',state.page*state.size); }
  return p;
 }
@@ -321,14 +326,14 @@ function select(id,auto){
    <dt>Result</dt><dd><span class="pill ${r.overall_result}">${esc(r.overall_result)}</span></dd>
    <dt>Log</dt><dd>${esc(r.log_type)}</dd>
    ${r.work_order?`<dt>WO</dt><dd>${esc(r.work_order)}</dd>`:''}
-    <dt>Web</dt><dd>${r.api_uploaded_at?'published '+fmtDate(r.api_uploaded_at):'not published'}</dd></dl>`;
+    <dt>Web</dt><dd>${r.api_uploaded_at?`<span class="tag pub">Published</span> <span style="color:var(--ink-3)">${fmtDate(r.api_uploaded_at)}</span>`:'<span class="tag unpub">Not published</span>'}</dd></dl>`;
  const ds=API+'/api/datasheet/'+id;
  $('acts').style.display='flex';
  $('acts').innerHTML=`<a class="pri" href="${ds}?format=html" target="_blank">Open ↗</a>
    <button onclick="printCert()">Print</button>
    <a href="${ds}?format=txt" download="${esc(r.serial_number)}.txt">TXT</a>
    <a href="${ds}?format=html" download="${esc(r.serial_number)}.html">HTML</a>
-    <button disabled title="Re-publish this cert — needs a POST /api/publish endpoint">Push to Web ▴</button>`;
+    <button onclick="pushWeb('${id}',this)" title="Publish / re-publish this cert to the public website">Push to Web ▴</button>`;
  $('insp').classList.add('open');
  // lazy-load the certificate so fast arrow/typing stays snappy
  $('viewer').innerHTML='<div class="state"><div class="skel" style="width:60%"></div></div>';
@@ -338,18 +343,23 @@ function select(id,auto){
 }
 function loadCert(ds){
  $('viewer').innerHTML='<iframe id="dsframe" title="datasheet"></iframe>';
-  const f=$('dsframe'); f.onload=()=>{styleCert();fitCert();}; f.src=ds+'?format=html';
+  const f=$('dsframe'); f.onload=()=>{styleCert();fitCert();setTimeout(fitCert,120);setTimeout(fitCert,350);}; f.src=ds+'?format=html';
 }
 function styleCert(){ const f=$('dsframe'); try{ const doc=f.contentDocument; if(!doc)return;
  if(!doc.getElementById('_inj')){ const s=doc.createElement('style'); s.id='_inj';
-    s.textContent='html,body{background:#fff!important;margin:0!important}body{padding:22px 26px!important;color:#0f172a}pre{margin:0;font-family:'+getComputedStyle(document.body).getPropertyValue('--mono')+';font-size:12.5px;line-height:1.32}';
+    s.textContent='html,body{background:#fff!important;margin:0!important}body{padding:16px 20px!important;color:#0f172a}pre{margin:0;font-family:'+getComputedStyle(document.body).getPropertyValue('--mono')+';font-size:12.5px;line-height:1.32}';
    (doc.head||doc.documentElement).appendChild(s); }
 }catch(e){} }
-function fitCert(){ const f=$('dsframe'); if(!f)return; try{ const doc=f.contentDocument; if(!doc)return;
-  const root=doc.documentElement; root.style.zoom='';
-  const nat=Math.max(doc.body?doc.body.scrollWidth:0,root.scrollWidth), av=f.clientWidth-2;
-  root.style.zoom=(nat>av)?Math.max(.45,av/nat):1;
-  f.style.height=Math.ceil((doc.body?doc.body.scrollHeight:600)*(nat>av?av/nat:1)+4)+'px';
+function fitCert(){ const f=$('dsframe'); if(!f)return; try{ const doc=f.contentDocument; if(!doc||!doc.body)return;
+  const b=doc.body, root=doc.documentElement;
+  // measure natural content width (transform doesn't reflow, so text never rewraps)
+  b.style.transform='none'; b.style.width='max-content'; b.style.transformOrigin='0 0';
+  root.style.overflow='hidden';
+  const nat=Math.max(b.scrollWidth,root.scrollWidth), av=f.clientWidth-12;   // widest line + right margin
+  if(!nat){ return; }                                   // not laid out yet — the retry will catch it
+  const k=nat>av ? Math.max(.4, av/nat) : 1;
+  b.style.transform='scale('+k+')';
+  f.style.height=Math.ceil(b.scrollHeight*k+2)+'px';     // size the frame to the scaled cert, no v-scroll-in-frame
 }catch(e){} }
 function printCert(){ const f=$('dsframe'); if(f&&f.contentWindow){f.contentWindow.focus();f.contentWindow.print();} }
 window.addEventListener('resize',()=>{fitCert();});
@@ -363,6 +373,31 @@ $('copySel').onclick=()=>{ const sns=state.rows.filter(r=>state.checks.has(Strin
  navigator.clipboard&&navigator.clipboard.writeText(sns); $('copySel').textContent='Copied ✓'; setTimeout(()=>$('copySel').textContent='Copy serials',1200); };
 $('selclear').onclick=()=>{ state.checks.clear(); [...$('rows').querySelectorAll('[data-ck]')].forEach(c=>c.checked=false); $('ckAll').checked=false; updateSel(); };

+/* ---------- publish to public website (POST /api/upload, idempotent) ---------- */
+async function doUpload(payload){
+  const r=await fetch(API+'/api/upload',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(payload)});
+  const d=await r.json().catch(()=>({})); if(!r.ok) throw new Error(d.error||('HTTP '+r.status)); return d;
+}
+async function pushWeb(id,btn){
+  const r=state.rows.find(x=>x.id==id); const sn=r?r.serial_number:id;
+  if(!confirm('Publish '+sn+' to the public Dataforth website now?')) return;
+  const t=btn.textContent; btn.disabled=true; btn.textContent='Publishing…';
+  try{ const d=await doUpload({ids:[+id]});
+    btn.textContent=d.errors?('✕ '+d.errors+' err'):(d.skipped&&!(d.created+d.updated+d.unchanged)?'skipped':'Published ✓');
+    setTimeout(search,500);                                  // refresh WEB status from the DB
+  }catch(e){ btn.textContent='✕ '+e.message.slice(0,16); btn.disabled=false; }
+}
+async function pushSelected(){
+  const ids=[...state.checks].map(Number); if(!ids.length) return;
+  if(!confirm('Publish '+ids.length+' selected serial(s) to the public Dataforth website now?')) return;
+  const b=$('repushSel'),t=b.textContent; b.disabled=true; b.textContent='Publishing…';
+  try{ const d=await doUpload({ids});
+    b.textContent='✓ '+((d.created||0)+(d.updated||0))+' pushed'+(d.skipped?(' · '+d.skipped+' skip'):'');
+    setTimeout(()=>{b.textContent=t;b.disabled=false;search();},1400);
+  }catch(e){ b.textContent='✕ failed'; b.disabled=false; alert('Push failed: '+e.message); }
+}
+$('repushSel').onclick=pushSelected;
+
 /* ---------- sort ---------- */
 function updateSortHeads(){ [...document.querySelectorAll('thead th.s')].forEach(th=>{
  const on=th.dataset.s===state.sort; th.querySelector('.arr')?.remove();
@@ -385,12 +420,12 @@ $('pageSize').onchange=e=>{state.size=+e.target.value;state.page=0;search();};
 $('prev').onclick=()=>{if(state.page>0){state.page--;search();$('twrap').scrollTop=0;}};
 $('next').onclick=()=>{state.page++;search();$('twrap').scrollTop=0;};
 $('reset').onclick=()=>{ ['serial','model','q','result','station','logtype','from','to'].forEach(k=>state[k]='');
-  state.sort='';state.checks.clear();updateSel(); $('omni').value='';$('mode').textContent=state.force||'auto';
+  state.sort='';state.webStatus='';state.checks.clear();updateSel(); $('omni').value='';$('mode').textContent=state.force||'auto';
  $('fFrom').value=$('fTo').value=$('fStation').value=$('fLog').value=$('fModel').value=''; state.page=0; search(); };
 $('menu').onclick=()=>document.body.classList.toggle('rail-open');

 /* ---------- presets ---------- */
-function clearAll(){ ['serial','model','q','result','station','logtype','from','to'].forEach(k=>state[k]='');
+function clearAll(){ ['serial','model','q','result','station','logtype','from','to'].forEach(k=>state[k]=''); state.webStatus='';state.sort='';
  $('omni').value='';$('mode').textContent=state.force||'auto';$('fFrom').value=$('fTo').value=$('fStation').value=$('fLog').value=$('fModel').value=''; }
 function applyPreset(fn){ clearAll(); fn(); state.page=0; state.checks.clear(); updateSel(); document.body.classList.remove('rail-open'); search(); }
 const PRESETS=[
@@ -398,11 +433,11 @@ const PRESETS=[
  {ic:'✕',label:'Failures',fn:()=>{state.result='FAIL';}},
  {ic:'•',label:'Today',fn:()=>{const t=isoD(new Date());state.from=t;state.to=t;}},
  {ic:'7',label:'Last 7 days',fn:()=>{const d=new Date();d.setDate(d.getDate()-7);state.from=isoD(d);}},
+  {ic:'▴',label:'Latest uploads',fn:()=>{state.webStatus='on';state.sort='api_uploaded_at';state.dir='desc';}},
+  {ic:'○',label:'Not yet published',fn:()=>{state.webStatus='off';}},
 ];
 const SOON=[
-  {label:'Latest upload batch',why:'needs sort=uploaded in /api/search'},
-  {label:'Retested units',why:'needs a retest flag in the pipeline'},
-  {label:'Not yet published',why:'needs a published filter in /api/search'},
+  {label:'Retested units',why:'needs a retest flag in the pipeline (next)'},
 ];
 function renderPresets(){
  $('presets').innerHTML='';
--- a/projects/dataforth-dos/testdatadb-fix/routes/api.js
+++ b/projects/dataforth-dos/testdatadb-fix/routes/api.js
@@ -1,27 +1,17 @@
 /**
 * API Routes for Test Data Database
 *
- * Fixed version - uses a single persistent database connection instead of
- * opening and closing on every request. WAL journal mode enabled for
- * concurrent read support. Limit parameter capped at 1000.
+ * PostgreSQL version - uses pg.Pool via database/db.js.
+ * All route handlers are async. FTS uses tsvector/plainto_tsquery.
 */

 const express = require('express');
 const path = require('path');
-const Database = require('better-sqlite3');
+const db = require('../database/db');
 const { generateDatasheet } = require('../templates/datasheet');

 const router = express.Router();

-// ---------------------------------------------------------------------------
-// Singleton database connection - opened once at module load
-// ---------------------------------------------------------------------------
-const DB_PATH = path.join(__dirname, '..', 'database', 'testdata.db');
-
-const db = new Database(DB_PATH, { readonly: false });
-db.pragma('journal_mode = WAL');
-db.pragma('busy_timeout = 5000');
-
 // ---------------------------------------------------------------------------
 // Helpers
 // ---------------------------------------------------------------------------
@@ -42,107 +32,88 @@ function clampOffset(value) {
 // ---------------------------------------------------------------------------
 // GET /api/search
 // Search test records
-// Query params: serial, model, from, to, result, q, station, logtype, limit, offset
+// Query params: serial, model, from, to, result, q, station, logtype, web_status, limit, offset
 // ---------------------------------------------------------------------------
-router.get('/search', (req, res) => {
+router.get('/search', async (req, res) => {
    try {
-        const { serial, model, from, to, result, q, station, logtype } = req.query;
+        const { serial, model, from, to, result, q, station, logtype, workorder, web_status } = req.query;
        const limit = clampLimit(req.query.limit || 100);
        const offset = clampOffset(req.query.offset || 0);

-        let sql = 'SELECT * FROM test_records WHERE 1=1';
+        const conditions = [];
        const params = [];
+        let paramIdx = 0;
+
+        const addParam = (val) => {
+            paramIdx++;
+            params.push(val);
+            return '$' + paramIdx;
+        };
+
+        if (q) {
+            // Full-text search using tsvector
+            conditions.push(`search_vector @@ plainto_tsquery('english', ${addParam(q)})`);
+        }

        if (serial) {
-            sql += ' AND serial_number LIKE ?';
-            params.push(serial.includes('%') ? serial : `%${serial}%`);
+            const val = serial.includes('%') ? serial : `%${serial}%`;
+            conditions.push(`serial_number LIKE ${addParam(val)}`);
+        }
+
+        if (workorder) {
+            conditions.push(`work_order = ${addParam(workorder)}`);
        }

        if (model) {
-            sql += ' AND model_number LIKE ?';
-            params.push(model.includes('%') ? model : `%${model}%`);
+            const val = model.includes('%') ? model : `%${model}%`;
+            conditions.push(`model_number LIKE ${addParam(val)}`);
        }

        if (from) {
-            sql += ' AND test_date >= ?';
-            params.push(from);
+            conditions.push(`test_date >= ${addParam(from)}`);
        }

        if (to) {
-            sql += ' AND test_date <= ?';
-            params.push(to);
+            conditions.push(`test_date <= ${addParam(to)}`);
        }

        if (result) {
-            sql += ' AND overall_result = ?';
-            params.push(result.toUpperCase());
+            conditions.push(`overall_result = ${addParam(result.toUpperCase())}`);
        }

        if (station) {
-            sql += ' AND test_station = ?';
-            params.push(station);
+            conditions.push(`test_station = ${addParam(station)}`);
        }

        if (logtype) {
-            sql += ' AND log_type = ?';
-            params.push(logtype);
+            conditions.push(`log_type = ${addParam(logtype)}`);
        }

-        if (q) {
-            // Full-text search - rebuild query with FTS
-            sql = `SELECT test_records.* FROM test_records
-                   JOIN test_records_fts ON test_records.id = test_records_fts.rowid
-                   WHERE test_records_fts MATCH ?`;
-            params.length = 0;
-            params.push(q);
-
-            if (serial) {
-                sql += ' AND serial_number LIKE ?';
-                params.push(serial.includes('%') ? serial : `%${serial}%`);
-            }
-            if (model) {
-                sql += ' AND model_number LIKE ?';
-                params.push(model.includes('%') ? model : `%${model}%`);
-            }
-            if (station) {
-                sql += ' AND test_station = ?';
-                params.push(station);
-            }
-            if (logtype) {
-                sql += ' AND log_type = ?';
-                params.push(logtype);
-            }
-            if (result) {
-                sql += ' AND overall_result = ?';
-                params.push(result.toUpperCase());
-            }
-            if (from) {
-                sql += ' AND test_date >= ?';
-                params.push(from);
-            }
-            if (to) {
-                sql += ' AND test_date <= ?';
-                params.push(to);
-            }
+        if (req.query.web_status === 'off') {
+            conditions.push('api_uploaded_at IS NULL');
+        } else if (req.query.web_status === 'on') {
+            conditions.push('api_uploaded_at IS NOT NULL');
        }

-        sql += ' ORDER BY test_date DESC, serial_number';
-        sql += ' LIMIT ? OFFSET ?';
-        params.push(limit, offset);
+        const where = conditions.length > 0 ? 'WHERE ' + conditions.join(' AND ') : '';

-        const records = db.prepare(sql).all(...params);
+        // whitelisted sort (prevents injection); NULLS LAST so e.g. unpublished rows don't lead an api_uploaded_at sort
+        const SORTABLE = { serial_number:'serial_number', model_number:'model_number', test_date:'test_date', overall_result:'overall_result', test_station:'test_station', log_type:'log_type', api_uploaded_at:'api_uploaded_at' };
+        const sortCol = SORTABLE[req.query.sort];
+        const sortDir = String(req.query.dir || '').toLowerCase() === 'asc' ? 'ASC' : 'DESC';
+        const orderBy = sortCol ? `ORDER BY ${sortCol} ${sortDir} NULLS LAST, serial_number` : 'ORDER BY test_date DESC, serial_number';
+        const dataSql = `SELECT * FROM test_records ${where} ${orderBy} LIMIT ${addParam(limit)} OFFSET ${addParam(offset)}`;
+        const countSql = `SELECT COUNT(*) as count FROM test_records ${where}`;
+        const countParams = params.slice(0, paramIdx - 2); // exclude limit/offset

-        // Get total count
-        let countSql = sql.replace(/SELECT .* FROM/, 'SELECT COUNT(*) as count FROM')
-                         .replace(/ORDER BY.*$/, '');
-        countSql = countSql.replace(/LIMIT \? OFFSET \?/, '');
-
-        const countParams = params.slice(0, -2);
-        const total = db.prepare(countSql).get(...countParams);
+        const [records, countRow] = await Promise.all([
+            db.query(dataSql, params),
+            db.queryOne(countSql, countParams),
+        ]);

        res.json({
            records,
-            total: total?.count || records.length,
+            total: countRow?.count ? parseInt(countRow.count, 10) : records.length,
            limit,
            offset
        });
@@ -156,9 +127,9 @@ router.get('/search', (req, res) => {
 // GET /api/record/:id
 // Get single record by ID
 // ---------------------------------------------------------------------------
-router.get('/record/:id', (req, res) => {
+router.get('/record/:id', async (req, res) => {
    try {
-        const record = db.prepare('SELECT * FROM test_records WHERE id = ?').get(req.params.id);
+        const record = await db.queryOne('SELECT * FROM test_records WHERE id = $1', [req.params.id]);

        if (!record) {
            return res.status(404).json({ error: 'Record not found' });
@@ -176,21 +147,102 @@ router.get('/record/:id', (req, res) => {
 // Generate datasheet for a record
 // Query params: format (html, txt)
 // ---------------------------------------------------------------------------
-router.get('/datasheet/:id', (req, res) => {
+router.get('/datasheet/:id', async (req, res) => {
    try {
-        const record = db.prepare('SELECT * FROM test_records WHERE id = ?').get(req.params.id);
+        const record = await db.queryOne('SELECT * FROM test_records WHERE id = $1', [req.params.id]);

        if (!record) {
            return res.status(404).json({ error: 'Record not found' });
        }

        const format = req.query.format || 'html';
-        const datasheet = generateDatasheet(record, format);

-        if (format === 'html') {
-            res.type('html').send(datasheet);
+        // Try exact-match formatter first
+        const { loadAllSpecs, getSpecs } = require('../parsers/spec-reader');
+        const { generateExactDatasheet } = require('../templates/datasheet-exact');
+        const specMap = loadAllSpecs();
+        const specs = getSpecs(specMap, record.model_number);
+        const exactTxt = generateExactDatasheet(record, specs);
+
+        if (exactTxt && format === 'html') {
+            // Render exact-match TXT as styled HTML page
+            const escaped = exactTxt
+                .replace(/&/g, '&amp;')
+                .replace(/</g, '&lt;')
+                .replace(/>/g, '&gt;');
+            const html = `<!DOCTYPE html>
+<html>
+<head>
+    <title>Test Data Sheet - ${record.serial_number}</title>
+    <style>
+        body {
+            margin: 0;
+            padding: 20px;
+            background: #f0f0f0;
+            display: flex;
+            justify-content: center;
+        }
+        .page {
+            background: white;
+            padding: 40px 30px;
+            max-width: 720px;
+            width: 100%;
+            box-shadow: 0 2px 8px rgba(0,0,0,0.15);
+            border: 1px solid #ccc;
+        }
+        pre {
+            font-family: 'Courier New', Courier, monospace;
+            font-size: 11px;
+            line-height: 1.4;
+            margin: 0;
+            white-space: pre;
+            overflow-x: auto;
+        }
+        .toolbar {
+            position: fixed;
+            top: 10px;
+            right: 10px;
+            display: flex;
+            gap: 8px;
+        }
+        .toolbar button {
+            padding: 8px 16px;
+            border: 1px solid #999;
+            background: white;
+            cursor: pointer;
+            font-size: 13px;
+            border-radius: 4px;
+        }
+        .toolbar button:hover { background: #e0e0e0; }
+        @media print {
+            body { background: white; padding: 0; }
+            .page { box-shadow: none; border: none; padding: 0; }
+            .toolbar { display: none; }
+        }
+    </style>
+</head>
+<body>
+    <div class="toolbar">
+        <button onclick="window.print()">Print</button>
+        <button onclick="window.open('/api/datasheet/${record.id}/pdf')">Download PDF</button>
+        <button onclick="window.close()">Close</button>
+    </div>
+    <div class="page">
+        <pre>${escaped}</pre>
+    </div>
+</body>
+</html>`;
+            res.type('html').send(html);
+        } else if (exactTxt && format === 'txt') {
+            res.type('text/plain').send(exactTxt);
        } else {
-            res.type('text/plain').send(datasheet);
+            // Fall back to generic template
+            const datasheet = generateDatasheet(record, format);
+            if (format === 'html') {
+                res.type('html').send(datasheet);
+            } else {
+                res.type('text/plain').send(datasheet);
+            }
        }
    } catch (err) {
        console.error(`[${new Date().toISOString()}] [DATASHEET ERROR] ${err.message}`);
@@ -198,45 +250,83 @@ router.get('/datasheet/:id', (req, res) => {
    }
 });

+// ---------------------------------------------------------------------------
+// GET /api/datasheet/:id/pdf
+// Generate PDF datasheet for a record (on-demand download)
+// ---------------------------------------------------------------------------
+router.get('/datasheet/:id/pdf', async (req, res) => {
+    try {
+        const record = await db.queryOne('SELECT * FROM test_records WHERE id = $1', [req.params.id]);
+
+        if (!record) {
+            return res.status(404).json({ error: 'Record not found' });
+        }
+
+        const { loadAllSpecs, getSpecs } = require('../parsers/spec-reader');
+        const { generateExactDatasheet } = require('../templates/datasheet-exact');
+        const PDFDocument = require('pdfkit');
+
+        const specMap = loadAllSpecs();
+        const specs = getSpecs(specMap, record.model_number);
+        let txt = generateExactDatasheet(record, specs);
+
+        // Fall back to generic datasheet if exact-match formatter doesn't support this family
+        if (!txt) {
+            txt = generateDatasheet(record, 'txt');
+        }
+
+        if (!txt) {
+            return res.status(422).json({ error: 'Could not generate datasheet (missing specs or data)' });
+        }
+
+        const doc = new PDFDocument({
+            size: 'LETTER',
+            margins: { top: 36, bottom: 36, left: 36, right: 36 }
+        });
+
+        res.setHeader('Content-Type', 'application/pdf');
+        res.setHeader('Content-Disposition', `attachment; filename="${record.serial_number}.pdf"`);
+        doc.pipe(res);
+
+        doc.font('Courier').fontSize(9.5);
+        const lines = txt.split(/\r?\n/);
+        for (const line of lines) {
+            doc.text(line, { lineGap: 1 });
+        }
+
+        doc.end();
+    } catch (err) {
+        console.error(`[${new Date().toISOString()}] [PDF ERROR] ${err.message}`);
+        res.status(500).json({ error: err.message });
+    }
+});
+
 // ---------------------------------------------------------------------------
 // GET /api/stats
 // Get database statistics
 // ---------------------------------------------------------------------------
-router.get('/stats', (req, res) => {
+router.get('/stats', async (req, res) => {
    try {
-        const stats = {
-            total_records: db.prepare('SELECT COUNT(*) as count FROM test_records').get().count,
-            by_log_type: db.prepare(`
-                SELECT log_type, COUNT(*) as count
-                FROM test_records
-                GROUP BY log_type
-                ORDER BY count DESC
-            `).all(),
-            by_result: db.prepare(`
-                SELECT overall_result, COUNT(*) as count
-                FROM test_records
-                GROUP BY overall_result
-            `).all(),
-            by_station: db.prepare(`
-                SELECT test_station, COUNT(*) as count
-                FROM test_records
-                WHERE test_station IS NOT NULL AND test_station != ''
-                GROUP BY test_station
-                ORDER BY test_station
-            `).all(),
-            date_range: db.prepare(`
-                SELECT MIN(test_date) as oldest, MAX(test_date) as newest
-                FROM test_records
-            `).get(),
-            recent_serials: db.prepare(`
-                SELECT DISTINCT serial_number, model_number, test_date
-                FROM test_records
-                ORDER BY test_date DESC
-                LIMIT 10
-            `).all()
-        };
+        const [totalRow, byLogType, byResult, byStation, dateRange, recentSerials] = await Promise.all([
+            db.queryOne('SELECT COUNT(*) as count FROM test_records'),
+            db.query('SELECT log_type, COUNT(*) as count FROM test_records GROUP BY log_type ORDER BY count DESC'),
+            db.query('SELECT overall_result, COUNT(*) as count FROM test_records GROUP BY overall_result'),
+            db.query(`SELECT test_station, COUNT(*) as count FROM test_records
+                      WHERE test_station IS NOT NULL AND test_station != ''
+                      GROUP BY test_station ORDER BY test_station`),
+            db.queryOne('SELECT MIN(test_date) as oldest, MAX(test_date) as newest FROM test_records'),
+            db.query(`SELECT DISTINCT serial_number, model_number, test_date
+                      FROM test_records ORDER BY test_date DESC LIMIT 10`),
+        ]);

-        res.json(stats);
+        res.json({
+            total_records: parseInt(totalRow.count, 10),
+            by_log_type: byLogType.map(r => ({ ...r, count: parseInt(r.count, 10) })),
+            by_result: byResult.map(r => ({ ...r, count: parseInt(r.count, 10) })),
+            by_station: byStation.map(r => ({ ...r, count: parseInt(r.count, 10) })),
+            date_range: dateRange,
+            recent_serials: recentSerials,
+        });
    } catch (err) {
        console.error(`[${new Date().toISOString()}] [STATS ERROR] ${err.message}`);
        res.status(500).json({ error: err.message });
@@ -247,30 +337,22 @@ router.get('/stats', (req, res) => {
 // GET /api/filters
 // Get available filter options (test stations, log types, models)
 // ---------------------------------------------------------------------------
-router.get('/filters', (req, res) => {
+router.get('/filters', async (req, res) => {
    try {
-        const filters = {
-            stations: db.prepare(`
-                SELECT DISTINCT test_station
-                FROM test_records
-                WHERE test_station IS NOT NULL AND test_station != ''
-                ORDER BY test_station
-            `).all().map(r => r.test_station),
-            log_types: db.prepare(`
-                SELECT DISTINCT log_type
-                FROM test_records
-                ORDER BY log_type
-            `).all().map(r => r.log_type),
-            models: db.prepare(`
-                SELECT DISTINCT model_number, COUNT(*) as count
-                FROM test_records
-                GROUP BY model_number
-                ORDER BY count DESC
-                LIMIT 500
-            `).all()
-        };
+        const [stations, logTypes, models] = await Promise.all([
+            db.query(`SELECT DISTINCT test_station FROM test_records
+                      WHERE test_station IS NOT NULL AND test_station != ''
+                      ORDER BY test_station`),
+            db.query('SELECT DISTINCT log_type FROM test_records ORDER BY log_type'),
+            db.query(`SELECT DISTINCT model_number, COUNT(*) as count FROM test_records
+                      GROUP BY model_number ORDER BY count DESC LIMIT 500`),
+        ]);

-        res.json(filters);
+        res.json({
+            stations: stations.map(r => r.test_station),
+            log_types: logTypes.map(r => r.log_type),
+            models: models.map(r => ({ ...r, count: parseInt(r.count, 10) })),
+        });
    } catch (err) {
        console.error(`[${new Date().toISOString()}] [FILTERS ERROR] ${err.message}`);
        res.status(500).json({ error: err.message });
@@ -281,51 +363,60 @@ router.get('/filters', (req, res) => {
 // GET /api/export
 // Export search results as CSV
 // ---------------------------------------------------------------------------
-router.get('/export', (req, res) => {
+router.get('/export', async (req, res) => {
    try {
        const { serial, model, from, to, result, station, logtype } = req.query;

-        let sql = 'SELECT * FROM test_records WHERE 1=1';
+        const conditions = [];
        const params = [];
+        let paramIdx = 0;
+
+        const addParam = (val) => {
+            paramIdx++;
+            params.push(val);
+            return '$' + paramIdx;
+        };

        if (serial) {
-            sql += ' AND serial_number LIKE ?';
-            params.push(serial.includes('%') ? serial : `%${serial}%`);
+            const val = serial.includes('%') ? serial : `%${serial}%`;
+            conditions.push(`serial_number LIKE ${addParam(val)}`);
        }

        if (model) {
-            sql += ' AND model_number LIKE ?';
-            params.push(model.includes('%') ? model : `%${model}%`);
+            const val = model.includes('%') ? model : `%${model}%`;
+            conditions.push(`model_number LIKE ${addParam(val)}`);
        }

        if (from) {
-            sql += ' AND test_date >= ?';
-            params.push(from);
+            conditions.push(`test_date >= ${addParam(from)}`);
        }

        if (to) {
-            sql += ' AND test_date <= ?';
-            params.push(to);
+            conditions.push(`test_date <= ${addParam(to)}`);
        }

        if (result) {
-            sql += ' AND overall_result = ?';
-            params.push(result.toUpperCase());
+            conditions.push(`overall_result = ${addParam(result.toUpperCase())}`);
        }

        if (station) {
-            sql += ' AND test_station = ?';
-            params.push(station);
+            conditions.push(`test_station = ${addParam(station)}`);
        }

        if (logtype) {
-            sql += ' AND log_type = ?';
-            params.push(logtype);
+            conditions.push(`log_type = ${addParam(logtype)}`);
        }

-        sql += ' ORDER BY test_date DESC, serial_number LIMIT 10000';
+        if (req.query.web_status === 'off') {
+            conditions.push('api_uploaded_at IS NULL');
+        } else if (req.query.web_status === 'on') {
+            conditions.push('api_uploaded_at IS NOT NULL');
+        }

-        const records = db.prepare(sql).all(...params);
+        const where = conditions.length > 0 ? 'WHERE ' + conditions.join(' AND ') : '';
+        const sql = `SELECT * FROM test_records ${where} ORDER BY test_date DESC, serial_number LIMIT 10000`;
+
+        const records = await db.query(sql, params);

        // Generate CSV
        const headers = ['id', 'log_type', 'model_number', 'serial_number', 'test_date', 'test_station', 'overall_result', 'source_file'];
@@ -348,16 +439,119 @@ router.get('/export', (req, res) => {
    }
 });

+// ---------------------------------------------------------------------------
+// GET /api/workorder/:wo
+// Get work order details and all associated test lines
+// ---------------------------------------------------------------------------
+router.get('/workorder/:wo', async (req, res) => {
+    try {
+        const wo = req.params.wo;
+
+        const [header, lines, testRecords] = await Promise.all([
+            db.queryOne('SELECT * FROM work_orders WHERE wo_number = $1', [wo]),
+            db.query('SELECT * FROM work_order_lines WHERE wo_number = $1 ORDER BY test_date, test_time', [wo]),
+            db.query(
+                'SELECT id, log_type, model_number, serial_number, test_date, test_station, overall_result, work_order FROM test_records WHERE work_order = $1 ORDER BY serial_number',
+                [wo]
+            ),
+        ]);
+
+        res.json({
+            work_order: header || { wo_number: wo },
+            lines,
+            test_records: testRecords,
+        });
+    } catch (err) {
+        console.error(`[${new Date().toISOString()}] [WO ERROR] ${err.message}`);
+        res.status(500).json({ error: err.message });
+    }
+});
+
+// ---------------------------------------------------------------------------
+// GET /api/workorder-search?q=<query>
+// Search work orders by number (prefix match)
+// ---------------------------------------------------------------------------
+router.get('/workorder-search', async (req, res) => {
+    try {
+        const q = req.query.q || '';
+        if (q.length < 2) {
+            return res.json({ results: [] });
+        }
+
+        const results = await db.query(
+            'SELECT wo_number, wo_date, program, test_station FROM work_orders WHERE wo_number LIKE $1 ORDER BY wo_date DESC LIMIT 50',
+            [q + '%']
+        );
+
+        res.json({ results });
+    } catch (err) {
+        res.status(500).json({ error: err.message });
+    }
+});
+
 // ---------------------------------------------------------------------------
 // Cleanup function for graceful shutdown
 // ---------------------------------------------------------------------------
-function cleanup() {
+async function cleanup() {
    try {
-        db.close();
+        await db.close();
    } catch (err) {
        console.error(`[${new Date().toISOString()}] [CLEANUP ERROR] ${err.message}`);
    }
 }

+/**
+ * POST /api/upload
+ *
+ * Body: { ids?: number[], serialNumbers?: string[], all_unuploaded?: boolean }
+ *
+ * Pushes selected records to the Dataforth website API. Accepts either a set
+ * of record IDs (resolved to serial_number + checked for exported status), a
+ * direct list of serial numbers, or all_unuploaded:true to push every PASS
+ * record where api_uploaded_at IS NULL.
+ *
+ * Response: { created, updated, unchanged, errors, skipped, processed, sns }
+ */
+router.post('/upload', async (req, res) => {
+    try {
+        const { ids, serialNumbers, all_unuploaded } = req.body || {};
+        const { uploadBySerialNumbers } = require('../database/upload-to-api');
+
+        let sns = [];
+        if (all_unuploaded) {
+            const rows = await db.query(
+                `SELECT DISTINCT serial_number FROM test_records
+                 WHERE overall_result = 'PASS'
+                   AND api_uploaded_at IS NULL
+                 ORDER BY serial_number`
+            );
+            sns = rows.map(r => r.serial_number);
+        } else if (Array.isArray(ids) && ids.length > 0) {
+            const placeholders = ids.map((_, i) => `$${i + 1}`).join(',');
+            const rows = await db.query(
+                `SELECT DISTINCT serial_number FROM test_records
+                 WHERE id IN (${placeholders})
+                   AND overall_result = 'PASS'`,
+                ids,
+            );
+            sns = rows.map(r => r.serial_number);
+        } else if (Array.isArray(serialNumbers) && serialNumbers.length > 0) {
+            sns = [...new Set(serialNumbers)];
+        } else {
+            return res.status(400).json({ error: 'provide ids[], serialNumbers[], or all_unuploaded=true' });
+        }
+
+        if (sns.length === 0) {
+            return res.json({ created:0, updated:0, unchanged:0, errors:0, skipped:0, processed:0, sns:[] });
+        }
+
+        const result = await uploadBySerialNumbers(sns);
+        res.json({ ...result, processed: sns.length, sns });
+    } catch (err) {
+        console.error(`[UPLOAD] ${err.message}`);
+        res.status(500).json({ error: err.message });
+    }
+});
+
 module.exports = router;
 module.exports.cleanup = cleanup;
--- a/projects/dataforth-dos/tools/mine-hoffman-dsca.py
+++ b/projects/dataforth-dos/tools/mine-hoffman-dsca.py
@@ -0,0 +1,123 @@
+#!/usr/bin/env python3
+"""
+Mine per-model DSCA33/DSCA45 Final-Test templates from the ORIGINAL certs stored
+on Dataforth's Hoffman API (the spec files lost in the cryptolocker event are
+recoverable here because the original software published these before the wipe).
+
+Input : a JSON map [{"m": model, "s": serial}, ...] of UPLOADED serials.
+Output: dsca33-45-templates.json  (schema-compatible with dsca-templates.json:
+        { model: { "accOut": "...", "rows": [ {"name","spec"}, ... ] } })
+        + a human report on stdout.
+
+Same extraction as the STAGE-1 extractor: the '===' rule under the Final-Test
+"Parameter ... Measured" header gives exact column spans; name = Parameter col,
+spec = Specification col. Keeps the richest sheet (most rows) per model.
+"""
+import json, re, sys, time, urllib.request, urllib.parse, os
+
+TOKEN_URL = "https://login.dataforth.com/connect/token"
+API_BASE  = "https://www.dataforth.com"
+CID, CSEC, SCOPE = "dataforth.onprem.sync", "Trxvwee2234-Awer8723-2", "dataforth.web"
+
+def get_token():
+    body = urllib.parse.urlencode({
+        "grant_type": "client_credentials", "client_id": CID,
+        "client_secret": CSEC, "scope": SCOPE}).encode()
+    req = urllib.request.Request(TOKEN_URL, body,
+        {"Content-Type": "application/x-www-form-urlencoded"})
+    return json.loads(urllib.request.urlopen(req, timeout=30).read())["access_token"]
+
+def get_cert(serial, tok):
+    url = f"{API_BASE}/api/v1/TestReportDataFiles/{urllib.parse.quote(serial)}"
+    req = urllib.request.Request(url, headers={"Authorization": f"Bearer {tok}"})
+    try:
+        with urllib.request.urlopen(req, timeout=30) as r:
+            return json.loads(r.read())
+    except urllib.error.HTTPError as e:
+        if e.code == 404: return None
+        raise
+
+def col_spans(sep):
+    return [(m.start(), m.end()) for m in re.finditer(r"=+", sep)]
+
+def extract(t):
+    lines = t.replace("\r\n", "\n").split("\n")
+    ahi = next((i for i, l in enumerate(lines)
+                if "Error (%)" in l and "Status" in l), -1)
+    acc_hdr = lines[ahi] if ahi >= 0 else ""
+    # capture the verbatim 2-line accuracy header (super-header + column line) so
+    # AD2 can reproduce the model-specific input label + VDC/mADC/Hz headers exactly
+    acc_header = [lines[ahi - 1].rstrip(), lines[ahi].rstrip()] if ahi > 0 else []
+    m = re.search(r"Output \([^)]*\)|Vout \([^)]*\)", acc_hdr)
+    acc_out = m.group(0) if m else "?"
+    fi = next((i for i, l in enumerate(lines) if "FINAL TEST RESULTS" in l), -1)
+    if fi < 0: return None
+    hi = next((i for i in range(fi + 1, len(lines))
+               if re.search(r"Parameter\s+Measured", lines[i])), -1)
+    if hi < 0: return None
+    sep = lines[hi + 1] if hi + 1 < len(lines) else ""
+    if "=" not in sep: return None
+    cols = col_spans(sep)
+    if len(cols) < 4: return None
+    pc, mc, sc, stc = cols[0], cols[1], cols[2], cols[3]
+    rows = []
+    for i in range(hi + 2, len(lines)):
+        l = lines[i]
+        if re.search(r"Check List|^\s*_{5,}", l): break
+        if not l.strip(): continue
+        name = l[pc[0]:mc[0]].strip()
+        spec = l[sc[0]:stc[0]].strip()
+        if not name and not spec: continue
+        rows.append({"name": name, "spec": spec})
+    return {"accOut": acc_out, "rows": rows, "accHdr": acc_hdr.strip(),
+            "accHeader": acc_header}
+
+def main():
+    mp = json.load(open(sys.argv[1]))
+    outpath = sys.argv[2]
+    tok = get_token()
+    by_model = {}     # model -> best {accOut, rows, accHdr, serial}
+    meta = {}         # model -> diagnostics
+    missing = []
+    for row in mp:
+        model, serial = row["m"], row["s"]
+        cert = get_cert(serial, tok)
+        if not cert or not cert.get("Content"):
+            missing.append((model, serial)); continue
+        tpl = extract(cert["Content"])
+        if not tpl:
+            meta.setdefault(model, {}).setdefault("noextract", []).append(serial); continue
+        cur = by_model.get(model)
+        if not cur or len(tpl["rows"]) > len(cur["rows"]):
+            tpl["serial"] = serial
+            by_model[model] = tpl
+    # build schema-compatible output
+    out = {}
+    for model in sorted(by_model):
+        t = by_model[model]
+        out[model] = {"accOut": t["accOut"], "accHeader": t["accHeader"],
+                      "rows": t["rows"], "_srcSerial": t["serial"]}
+    with open(outpath, "w") as f:
+        json.dump(out, f, indent=0)
+    # report
+    fams = {}
+    print(f"=== Mined {len(out)} models from Hoffman -> {outpath} ===\n")
+    print(f"{'MODEL':<14} {'rows':>4}  {'accOut':<16} src-serial   accuracy-header")
+    for model in sorted(out):
+        t = by_model[model]
+        fam = model.split("-")[0]
+        fams[fam] = fams.get(fam, 0) + 1
+        flag = "  <-- LOW" if len(t["rows"]) < 3 else ""
+        print(f"{model:<14} {len(t['rows']):>4}  {t['accOut']:<16} {t['serial']:<11}  {t['accHdr'][:60]}{flag}")
+    print("\nper-family models mined:", dict(fams))
+    distinct_accout = sorted(set(o["accOut"] for o in out.values()))
+    print("distinct accOut tokens:", distinct_accout)
+    if missing:
+        print(f"\n[WARN] {len(missing)} serials returned 404 (not on Hoffman):",
+              missing[:10], "..." if len(missing) > 10 else "")
+    no_tpl = [m for m in {r['m'] for r in mp} if m not in out]
+    if no_tpl:
+        print(f"\n[WARN] models with NO usable template ({len(no_tpl)}):", no_tpl)
+
+if __name__ == "__main__":
+    main()
--- a/projects/dataforth-dos/tools/preview-proxy.py
+++ b/projects/dataforth-dos/tools/preview-proxy.py
@@ -0,0 +1,71 @@
+#!/usr/bin/env python3
+"""
+Same-origin preview proxy for the testdatadb front-end.
+
+Serves the static prototype from ROOT and reverse-proxies /api/* to the live
+AD2 testdatadb server, so the app AND the cert iframe share one origin
+(http://127.0.0.1:PORT). That lets the same-origin-only cert styling/fit logic
+(styleCert/fitCert read iframe.contentDocument) actually run during preview —
+which it can't when the iframe is loaded cross-origin straight from AD2.
+
+Usage: python preview-proxy.py <port> <root-dir> [target]
+"""
+import http.server, socketserver, urllib.request, urllib.error, os, sys
+
+PORT = int(sys.argv[1])
+ROOT = os.path.abspath(sys.argv[2]) if len(sys.argv) > 2 else "."
+TARGET = sys.argv[3] if len(sys.argv) > 3 else "http://192.168.0.6:3000"
+
+class H(http.server.BaseHTTPRequestHandler):
+    def do_GET(self):
+        if self.path.startswith("/api/"):
+            try:
+                with urllib.request.urlopen(TARGET + self.path, timeout=30) as r:
+                    body = r.read(); ct = r.headers.get("Content-Type", "application/octet-stream")
+                self._send(200, ct, body)
+            except urllib.error.HTTPError as e:
+                self._send(e.code, "application/json", e.read())
+            except Exception as e:
+                self._send(502, "text/plain", str(e).encode())
+        else:
+            p = self.path.split("?")[0]
+            p = "/index.html" if p == "/" else p
+            fp = os.path.join(ROOT, p.lstrip("/"))
+            if os.path.isfile(fp):
+                ct = "text/html; charset=utf-8" if fp.endswith(".html") else "application/octet-stream"
+                self._send(200, ct, open(fp, "rb").read())
+            else:
+                self._send(404, "text/plain", b"not found")
+
+    def do_POST(self):
+        if self.path.startswith("/api/"):
+            n = int(self.headers.get("Content-Length", 0))
+            body = self.rfile.read(n) if n else b""
+            req = urllib.request.Request(
+                TARGET + self.path, data=body, method="POST",
+                headers={"Content-Type": self.headers.get("Content-Type", "application/json")})
+            try:
+                with urllib.request.urlopen(req, timeout=120) as r:
+                    self._send(200, r.headers.get("Content-Type", "application/json"), r.read())
+            except urllib.error.HTTPError as e:
+                self._send(e.code, "application/json", e.read())
+            except Exception as e:
+                self._send(502, "text/plain", str(e).encode())
+        else:
+            self._send(404, "text/plain", b"not found")
+
+    def _send(self, code, ct, body):
+        self.send_response(code)
+        self.send_header("Content-Type", ct)
+        self.send_header("Content-Length", str(len(body)))
+        self.send_header("Cache-Control", "no-store")
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, *a): pass
+
+class S(socketserver.ThreadingTCPServer):
+    allow_reuse_address = True
+
+print(f"preview proxy: http://127.0.0.1:{PORT}  root={ROOT}  -> {TARGET}")
+S(("127.0.0.1", PORT), H).serve_forever()
Author	SHA1	Message	Date
Mike Swanson	27170e1a5c	sync: auto-sync from GURU-5070 at 2026-06-18 12:49:38 Author: Mike Swanson Machine: GURU-5070 Timestamp: 2026-06-18 12:49:38	2026-06-18 12:51:08 -07:00
Mike Swanson	0745f5d09b	dataforth/testdatadb: 8B/5B/SCM render verify results + convergence plan Stage+verify (template-gated, no slotmaps/precision yet) vs Hoffman, content-only: 15 models content-perfect, 17 precision-distance, 70 NULL (need slotmaps), 8B38/7B family-specific. Remaining work = AD2's existing DSCA machinery (slotmaps / Math.fround QB rounding / frequency-AAC accuracy labels). Recommend converging with AD2's DSCA path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 12:50:44 -07:00
Mike Swanson	82ae349941	dataforth/testdatadb: mine 8B/5B/SCM Final-Test templates from Hoffman (136 models) Root cause of the ~5,148 unpublished 8B/5B/SCM PASS records (driving the fix): (1) parseRawData wrongly consumes a PASS/FAIL line as the step-response line for non-DSCA families that omit the "0","0",v line (8B45/8B49/5B39/SCM5B33...) -> drops the first Final-Test group -> measurement-count mismatch -> null render. (2) Even parsed, the renderer has ONE hardcoded DATA_LINES['8B'] (RTD-shaped), so models like 8B45 (frequency input, == DSCA45 structurally) get wrong param names/specs. Same class as DSCA -> needs per-model templates. Mined per-model templates from the Hoffman originals (published siblings) for all 136 mineable models via tools/mine-hoffman-dsca.py (family-agnostic extractor): 8b5bscm-templates.json = {accOut, accHeader, rows[name,spec], _srcSerial}. Input-type split: 72 voltage / 18 temp / 12 current (accuracy already handled) + 10 frequency (8B45/5B45 — same unsolved freq-accuracy as DSCA45) + 24 7B/other. Only 3 niche models (17 units) have no Hoffman original. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 12:50:44 -07:00
Mike Swanson	760c2264dc	dataforth/testdatadb: wire UI presets + publish buttons; add /api/search sort/dir Backend (deployed live on AD2, service restarted, + repo copy resynced — it was far behind the deployed server): - /api/search: add whitelisted sort/dir (NULLS LAST) so sortable headers and the "Latest uploads" preset work. web_status filter and POST /api/upload already existed on the server; the stale repo copy now matches live. Frontend (redesign prototype): - "Latest uploads" preset (web_status=on + sort=api_uploaded_at desc) and "Not yet published" (web_status=off) are now active presets. - Push to Web (inspector) + Re-push (multi-select) wired to POST /api/upload behind a confirm() gate; refresh WEB status after. Validated idempotently on a published record (unchanged:1, errors:0). - "Retested units" stays disabled — needs a retest flag in the pipeline (next). tools/preview-proxy.py: forward POST so the publish buttons work in same-origin preview. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 12:50:44 -07:00
Mike Swanson	72a2fbe6ce	dataforth/testdatadb UI: fix cert fit (transform-scale) + publish-state chips - fitCert: replace the flaky CSS `zoom` (Firefox support is recent/inconsistent) with transform:scale() measured against the widest line (+ right margin and font-load retries) so the cert always scales to fit the inspector with no horizontal clip. Validated live on a narrow 5B cert (0.74x) and a wide DSCA45 cert (0.55x) against the real AD2 dataset. - inspector Web field -> Published (green) / Not published (amber) chips. - widen default inspector 480 -> 500px. - tools/preview-proxy.py: serve the prototype AND reverse-proxy /api to the live AD2 server so the cert iframe is same-origin during preview — styleCert/fitCert read iframe.contentDocument, which silently no-ops when the iframe is loaded cross-origin straight from AD2 (why the fit looked broken in earlier previews). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 12:50:44 -07:00
Mike Swanson	419d6e58b5	dataforth/dsca33-45: recover lost specs from Hoffman API (56/58 models) The DSCA33/DSCA45 main spec files lost in the cryptolocker wipe are recoverable: the original software published correct certs to the Hoffman product API before the wipe and our null-skipping renderer never overwrote them. Mine per-model Final-Test templates (names + specs + verbatim accuracy headers) straight from those originals instead of requesting spec files from Dataforth/John. - dsca33-45-templates.json: 56 models (DSCA33 34/35, DSCA45 22/23); only DSCA33-1948 + DSCA45-1746 (24 units) lack an original. - mine-hoffman-dsca.py: the re-runnable miner. - DSCA33-45-HOFFMAN-RECOVERY handoff for the AD2 session (incl. the gate: validate each render vs its Hoffman original before enabling live rendering). - memories: Hoffman recovery (supersedes the spec-gap "need John" note) and the AD2 SSH MTU-blackhole root cause/fix; errorlog entries (syncro jq, ssh correction). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 12:50:43 -07:00