sync: auto-sync from AD2 at 2026-06-17 13:56:18
Author: Mike Swanson Machine: AD2 Timestamp: 2026-06-17 13:56:18
This commit is contained in:
@@ -0,0 +1,111 @@
|
||||
# 2026-06-17 — Dataforth datasheet RTD bug diagnosis + AD2 harness onboarding
|
||||
|
||||
## User
|
||||
- **User:** Mike Swanson (mike)
|
||||
- **Machine:** AD2
|
||||
- **Role:** admin
|
||||
|
||||
## Session Summary
|
||||
|
||||
Diagnosed a Dataforth test-datasheet defect reported 2026-06-17 by John Lehman / Peter Iliya, triggered by a Wellbore Integrity (Joseph Swinehart) cal-cert audit on 8B35 4-wire RTD certificates. The complaint: column headers wrong and some Final Test lines missing. Worked on AD2 (the testdatadb generator + PostgreSQL host), tracing the pipeline upstream from the website copy to the original ground-truth file. Diagnosis only — no changes to the generator or DB.
|
||||
|
||||
Confirmed two independent defects, both in the datasheet renderer `templates/datasheet-exact.js` (ingestion, DB data, and spec files are all correct). Defect A: RTD modules render the input column as resistance (`Rin (ohms)`) when the original/ground-truth file shows temperature (`Temp. (C)`); the stimulus values in `raw_data` are already temperatures (°C), so the numbers are right but the label is wrong and positive values lose their leading `+`. Defect B: the entire DSCA Final-Test parameter list is hardcoded as a single layout that does not match real DSCA module subtypes, producing wrong parameter names, garbage specs (`< 0 mA`, `+/- 0 %`), values mapped to the wrong rows, a mislabeled output column, and dropped lines. The "missing Final Test lines" complaint is a symptom of Defect B, not a separate bug.
|
||||
|
||||
Located the ground truth: the DOS test station writes a fully-rendered `.TXT` datasheet to `C:\STAGE\` before ingestion; it is mirrored to AD2 at `C:\Shares\test\STAGE\<TS>\<encoded-SN>.TXT`. Verified three examples against their originals — 8B35-04 (SN 179553-13, `H9553-13.TXT`), DSCA38-05 (SN 180224-7, `I0224-7.TXT`, a bridge module — NOT RTD), and DSCA34-05C (SN 180007-8, `I0007-8.TXT`, the actual DSCA RTD analog). Clarified for the thread that DSCA38 is a bridge/strain-gauge module, so its issue is Defect B; the DSCA RTD module Peter likely means is DSCA34, which has both A and B. Wrote the full diagnosis to `projects/dataforth-dos/DATASHEET-RTD-BUG-DIAGNOSIS-2026-06-17.md` (process doc + diagnosis + proposed fix).
|
||||
|
||||
The second half of the session was unplanned infrastructure work needed to `/sync` the results: AD2 had no Python (so `sync.sh` could not parse `identity.json`), no `identity.json` at all, and an unset git author identity. Installed Python 3.12.8, created a proper `identity.json` via the documented hand-create + `migrate-identity.sh` flow, and set git authorship. Then synced — which surfaced that `ad2` was 3 months behind main and that `sync.sh` cannot overwrite itself mid-run on Windows. Recovered cleanly, rebased `ad2` onto main (modernizing the fork to harness v1.4.3), relocated the Dataforth context out of the shared `.claude/CLAUDE.md` into `clients/dataforth/CLAUDE.dataforth.md` so future syncs stay conflict-free, and force-pushed.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- **Diagnose only, verify against the original file, not assumptions.** Pulled the staged DOS-generated `.TXT` as ground truth rather than trusting the code reading. This confirmed the RTD stimulus values are temperatures and the label is the defect.
|
||||
- **Classified the bug as renderer-only.** Ingestion (`multiline.js`), DB `raw_data`, and spec `SENTYPE` are all correct; the fix belongs in `datasheet-exact.js`. Defect A is surgical (fold RTD/sensorNum 7 into the temperature path); Defect B (DSCA) needs a per-subtype rebuild driven by `DSCFIN.DAT` and is larger.
|
||||
- **Installed Python from the official installer, not Chocolatey.** Choco 2.6.0 requires .NET 4.8 (Server 2019 ships 4.7.2); the python.org silent installer is self-contained and avoids a .NET install + reboot.
|
||||
- **Created identity.json via hand-create + migrate-identity.sh** (the documented flow) rather than inventing a generator. Used Mike's identity (strongly signaled by the gitea remote, branch, and user profile memory).
|
||||
- **Did not blind-commit the 205 MB WizTree zip until authorized.** `sync.sh` does `git add -A`; flagged the binary first. User confirmed AD2 is a Dataforth-ops fork and to sync everything.
|
||||
- **Modernized the ad2 fork by rebasing onto main and relocating the Dataforth doc.** Chose "take main's lean CLAUDE.md + move Dataforth context to a fork-specific file" so the fork stays additive and future rebases are conflict-free. The redundant CLAUDE.md-edit commit became empty and was auto-dropped.
|
||||
|
||||
## Problems Encountered
|
||||
|
||||
- **`sync.sh` aborted: "No Python interpreter found."** AD2 had no Python at all. Installed Python 3.12.8 (official amd64 installer, all-users, PATH + `py` launcher); `sync.sh` now detects `py`.
|
||||
- **`identity.json` missing** → commits would attribute as `unknown` and vault/coord fields were absent. Hand-created the core file and ran `migrate-identity.sh` to fill python/ollama/platform/arch/grok/coord_api. Validated all 13 required identity fields present; confirmed it is gitignored.
|
||||
- **Rebase failed mid-run: "unable to create file .claude/scripts/sync.sh: Permission denied."** `sync.sh` was the executing script while git's checkout-to-main tried to overwrite it (Windows file lock). The partial checkout left the working tree half-converted to origin/main (modified CLAUDE.md, deleted sync.sh, 240 untracked origin/main files). Recovered with `git reset --hard HEAD` + `git clean -fd`, then ran `git rebase origin/main` directly (outside sync.sh) — succeeded with only the expected `.claude/CLAUDE.md` conflict.
|
||||
- **CLAUDE.md rebase conflict** (Dataforth fork doc vs main's lean refactor). Resolved by taking main's version (`git checkout origin/main -- .claude/CLAUDE.md`); Dataforth content was preserved beforehand into `clients/dataforth/CLAUDE.dataforth.md`.
|
||||
- **NAS SSH host-key changed / SMB denied.** Could not reach `\\192.168.0.9\test\STAGE` over SSH (host key changed) or SMB (permission denied). Worked around via the rsync daemon (module `test`, user `rsync`) for listing, and found the staged originals already mirrored locally at `C:\Shares\test\STAGE\`.
|
||||
|
||||
## Configuration Changes
|
||||
|
||||
Created:
|
||||
- `C:\ClaudeTools\.claude\identity.json` (gitignored, per-machine) — full identity for AD2
|
||||
- `C:\ClaudeTools\projects\dataforth-dos\DATASHEET-RTD-BUG-DIAGNOSIS-2026-06-17.md` — diagnosis deliverable
|
||||
- `C:\ClaudeTools\clients\dataforth\CLAUDE.dataforth.md` — relocated Dataforth ops context
|
||||
- `C:\ClaudeTools\.claude\memory\project_ad2_dataforth_fork.md` (+ MEMORY.md index line)
|
||||
- Session log (this file)
|
||||
|
||||
Modified:
|
||||
- `.claude/CLAUDE.md` — now the lean fleet doc (was the Dataforth fork doc); rebased from main
|
||||
- Git author identity on AD2: `user.name="Mike Swanson"`, `user.email="mike@azcomputerguru.com"`
|
||||
- Whole harness advanced to main (v1.4.3): new `CLAUDE_EXTENDED.md`, `harness/`, `bootstrap/`, skills/commands
|
||||
|
||||
Installed:
|
||||
- Python 3.12.8 (64-bit, all-users) → `C:\Program Files\Python312\`, `py` launcher at `C:\Windows\py`
|
||||
|
||||
NOT changed (by design — diagnosis only): `templates/datasheet-exact.js`, the PostgreSQL DB, any datasheet content.
|
||||
|
||||
## Credentials & Secrets
|
||||
|
||||
No new credentials created or discovered this session. Used existing documented access:
|
||||
- NAS rsync daemon: `rsync://rsync@192.168.0.9/test` (module `test` = `/data/test`), password `IQ203s32119` (already in the Dataforth context doc).
|
||||
- PostgreSQL (local, AD2): default app creds from `database/db.js` — `testdatadb_app` / `DfTestDB2026!` on `localhost:5432/testdatadb`.
|
||||
|
||||
(AD2 has no SOPS vault cloned; these remain documented in `clients/dataforth/CLAUDE.dataforth.md`.)
|
||||
|
||||
## Infrastructure & Servers
|
||||
|
||||
- **AD2** 192.168.0.6 — testdatadb (Node/Express :3000) + PostgreSQL 18; this host. Hostname `AD2`.
|
||||
- **AD1** 192.168.0.27 — `\\AD1\Engineering`.
|
||||
- **D2TESTNAS** 192.168.0.9 — SMB1 bridge; rsync daemon port 873 module `test`; SSH host key changed this session (publickey/password denied).
|
||||
- testdatadb data: 464,671 records on website; log_types — 5BLOG 196,502 / 7BLOG 121,304 / DSCLOG 79,868 / 8BLOG 63,808 / others.
|
||||
- Impact of Defect A (RTD label): 8B35 5,476 + DSCA34 3,573 + SCM5B34/35 14,887 ≈ 24K certs. Defect B (DSCA template): up to 78,343 DSCLOG certs.
|
||||
|
||||
## Commands & Outputs
|
||||
|
||||
```bash
|
||||
# Render current generator output vs staged original (read-only diagnosis)
|
||||
cd C:\Shares\testdatadb
|
||||
node -e "const db=require('./database/db');const {renderContent}=require('./database/render-datasheet');(async()=>{const r=await db.queryOne('SELECT * FROM test_records WHERE serial_number=$1',['179553-13']);if(r.test_date&&r.test_date.toISOString)r.test_date=r.test_date.toISOString().slice(0,10);console.log(renderContent(r));await db.close();})()"
|
||||
type C:\Shares\test\STAGE\TS-4L\H9553-13.TXT # ground truth: header = " Temp. (C)"
|
||||
|
||||
# Python install (PowerShell)
|
||||
Start-Process python-3.12.8-amd64.exe -ArgumentList '/quiet','InstallAllUsers=1','PrependPath=1','Include_launcher=1' -Wait # exit 0
|
||||
|
||||
# Onboarding identity
|
||||
# hand-create .claude/identity.json (core fields) then:
|
||||
bash .claude/scripts/migrate-identity.sh # filled python/ollama/platform/arch/grok/coord_api
|
||||
|
||||
# Rebase recovery (sync.sh self-lock)
|
||||
git reset --hard HEAD && git clean -fd
|
||||
git rebase origin/main
|
||||
git checkout origin/main -- .claude/CLAUDE.md && git add .claude/CLAUDE.md
|
||||
GIT_EDITOR=true git rebase --continue
|
||||
git push --force-with-lease origin ad2
|
||||
```
|
||||
|
||||
Key error + resolution: `error: unable to create file .claude/scripts/sync.sh: Permission denied` → run the rebase directly (not via the executing sync.sh).
|
||||
|
||||
## Pending / Incomplete Tasks
|
||||
|
||||
- **Fix Defect A (RTD label/values)** in `templates/datasheet-exact.js`: in the input-header logic and `formatAccuracyLine`, route `sensorNum === 7` (RTD) through the temperature path (`' Temp. (C)'` + `formatSigned(stim, 2, 8)`). Verify leading-space alignment against a thermocouple original. Safe — `7` is reached only by RTD sentypes; no module currently needs `Rin (ohms)`. Awaiting review before changing the generator.
|
||||
- **Fix Defect B (DSCA template)**: rebuild the DSCA Final-Test parameter list + ACCURACY column titles/units per module subtype, driven by `specdata\DSCFIN.DAT` / the legacy QB DSC writer. Larger effort. Until done, treat all DSCA (DSCLOG) website datasheets as unreliable.
|
||||
- After fixes: re-push affected models by clearing `api_uploaded_at` (RE-PUSH is idempotent).
|
||||
- **AD2 tooling gaps**: `jq`, `sops`, `age` not installed; no vault cloned (`D:/vault` absent) → vault sync N/A. coord_api (172.16.3.30) unreachable from Dataforth LAN.
|
||||
- **`sync.sh` self-lock**: upstream fix candidate (re-exec from a temp copy, or rebase before the script can be overwritten) — offered to file as a CT thought.
|
||||
|
||||
## Reference Information
|
||||
|
||||
- Diagnosis doc: `projects/dataforth-dos/DATASHEET-RTD-BUG-DIAGNOSIS-2026-06-17.md`
|
||||
- Generator: `C:\Shares\testdatadb\templates\datasheet-exact.js` (repo: `projects/dataforth-dos/datasheet-pipeline/implementation/templates/datasheet-exact.js`)
|
||||
- Ground-truth originals: `C:\Shares\test\STAGE\<TS>\<encoded-SN>.TXT` (8.3 hex-prefix SN encoding: first 2 digits → letter, 55+n)
|
||||
- Examples: 8B35-04 SN 179553-13 (`H9553-13.TXT`, P1RTD4W, MAXIN 600); DSCA38-05 SN 180224-7 (`I0224-7.TXT`, FBRIDGE); DSCA34-05C SN 180007-8 (`I0007-8.TXT`, P1RTD3W)
|
||||
- Task prompt: `Prompt617.txt`
|
||||
- Commits this session (ad2): `49c9eb50` (work sync), `bbb19db2` (relocate Dataforth doc), `c4de16f6` (memory)
|
||||
- Contacts: John Lehman jlehman@dataforth.com, Peter Iliya pIliya@dataforth.com
|
||||
Reference in New Issue
Block a user