sync: auto-sync from GURU-5070 at 2026-07-21 08:42:56

Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-07-21 08:42:56

.claude/memory/MEMORY.md

@@ -27,6 +27,7 @@
 - [Radio Show Website](reference_radio_website.md) — Astro static site at radio.azcomputerguru.com on IX server.
 - [IX PHP-FPM/OPcache remediation](project_ix_php_fpm_opcache_remediation.md) — IX server default is suPHP/CGI per-request with OPcache cold: 71/72 sites recompile PHP every request (1-5.6s TTFB). Fix = per-vhost PHP-FPM. ezfast pilot done (3.1s->0.44s); fleet rollout proposed.
 - [Profile before hardware](feedback_profile_before_hardware.md) — For ANY performance complaint, measure/decompose the actual bottleneck BEFORE proposing hardware; config/cheapest cause first. (IX: $600 SSDs recommended before finding the real cause was OPcache-off — a config issue.)
+- [Dataforth datasheet pipeline recovery](project_dataforth_datasheet_pipeline_recovery.md) — 2026-07-21 recovery of DOS datasheet pipeline: NAS (D2TESTNAS, TS 100.85.152.90) DNS pointed at dead .1 (UDM is .254) → rsync reverse-DNS 10s timeouts stalled pulls; fixed. Daily task broke on PS mojibake parse error. import.js/test_date/_bulk_push_all internals. jlehman added to reports.
 - [IX Server Access](reference_ix_server_access.md) — `ix.azcomputerguru.com` / 172.16.3.10. Reachable when Tailscale is on (no VPN). SSH currently uses sshpass with root password; key auth from GURU-5070 not configured yet (was CachyOS, now Win11 — verify).
 - [Cloudflare access](reference_cloudflare_access.md) — Cloudflare API creds in SOPS `services/cloudflare.sops.yaml` (full DNS + account tokens; azcomputerguru zone_id 1beb9917...). azcomputerguru.com DNS is on Cloudflare (not IX) — edit via Cloudflare API, not whmapi1.
 - [Matomo Analytics](reference_matomo_analytics.md) — Self-hosted analytics at analytics.azcomputerguru.com, site IDs, tracking for all 3 sites.

.claude/memory/project_dataforth_datasheet_pipeline_recovery.md

@@ -0,0 +1,37 @@
+---
+name: project_dataforth_datasheet_pipeline_recovery
+description: Dataforth DOS datasheet pipeline "no new datasheets" recovery (2026-07-21) — NAS rsync reverse-DNS timeout + daily-task parse error; fixes + gotchas.
+metadata:
+  type: project
+---
+
+Dataforth test-datasheet pipeline stopped producing new datasheets ~2026-07-10. Full recovery 2026-07-21.
+
+**Root cause (transport):** D2TESTNAS (NAS, 192.168.0.9; Tailscale **100.85.152.90**, root+password, key auth
+denied; hostkey SHA256:DMMw8IJ4Aus54EAQOfJysLZ3Mw63WJ5R44ETiLMYeUI) had `resolv.conf nameserver 192.168.0.1`
+which is **DEAD** (100% packet loss). The **UDM is .254** (not .1). rsyncd does a reverse-DNS lookup per
+connection by default → every rsync connection ate a **10s DNS timeout** (`getent hosts` = 10.03s), so the
+per-station `.DAT` pulls (15-30s timeouts) failed "code 35" and no test data reached AD2. Fixes: (1) `reverse
+lookup = no` in `/etc/rsyncd.conf` + `systemctl restart rsync` (backstop; pull 10s->0.3s); (2) proper:
+`nmcli con mod "Wired connection 1" ipv4.dns 192.168.0.254; nmcli con mod ... ipv4.ignore-auto-dns yes; nmcli
+dev reapply eth0` (reverse lookup now 0.007s). NAS is Debian 13, static IPv4.
+
+**Second issue (daily report):** `DataforthTestDatasheetUploader` task (run-pipeline.ps1) failed since May 12 —
+a **PowerShell PARSE error from mojibake em-dashes** (lines 75/130/135/160 had corrupted `—`). Fix: ASCII-clean
+those lines (`-replace '[^\x00-\x7F]','-'`), verify with `[System.Management.Automation.PSParser]::Tokenize`.
+**GOTCHA: editing a .ps1 containing non-ASCII via `Get-Content -Raw` + `Set-Content -Encoding UTF8` double-
+encodes/corrupts the non-ASCII** — keep pipeline scripts ASCII-only; parse-test after any edit. See
+[[feedback_windows_quote_stripping]].
+
+**Pipeline internals learned:** import.js (`C:\Shares\testdatadb\database`) scans HISTLOGS + Recovery-TEST +
+station LOGS (`C:/Shares/test/TS-*/LOGS`) in **one transaction** — slow, commits only at the end; the sync
+(`Sync-FromNAS-rsync.ps1`, rsync daemon user `rsync`) pulls `.DAT` then calls it. `test_date` is a **TEXT**
+column (compare as ISO text, no `date` cast; `to_char` fails). `_bulk_push_all.js` pushes pending PASS to the
+Hoffman API (idempotent). Dashboard/API `:3000` (restart `testdatadb` service if wedged). Report recipients:
+`notify.js` `const TO` + `run-pipeline.ps1` `toRecipients` — **jlehman@dataforth.com added** 2026-07-21; Graph
+sendMail confirmed **202** (creds in `C:\ProgramData\dataforth-uploader\credentials.json` GRAPH_*).
+
+**Still open:** ~3,000 old **unrenderable** records (skipped on push — stale/missing specs, audit F4); the
+station-side inbound spec distribution (#32489) is separate. AD2 reached via GuruRMM agent
+(4b9fe588-...), not directly (192.168.0.x not on the Tailscale 172.16.0.0/22 route). Report:
+`clients/internal-infrastructure`? no — see clients/dataforth session log 2026-07-21. See [[project_ix_php_fpm_opcache_remediation]] for the same-session IX work.

clients/dataforth/session-logs/2026-07/2026-07-21-mike-datasheet-pipeline-outage-recovery.md

@@ -0,0 +1,121 @@
+## User
+- **User:** Mike Swanson (mike)
+- **Machine:** GURU-5070
+- **Role:** admin
+
+# Dataforth DOS Test-Datasheet Pipeline — Outage Recovery (John Lehman)
+
+**Date:** 2026-07-21 · **Client:** Dataforth Corporation · **Systems:** D2TESTNAS (NAS), AD2 (pipeline host)
+
+## Session Summary
+
+John Lehman reported the "DOS test flow isn't working" (no new datasheets on dataforth.com), asked to be
+included on the Success/Failure reports, and asked where to find a report. Loaded the Dataforth DOS project
+context (wiki + the 2026-07-01 ground-truth audit), then diagnosed live via the GuruRMM agent on AD2 (the box
+isn't directly routable — 192.168.0.x is not on the Tailscale 172.16.0.0/22 route).
+
+Ground truth: PostgreSQL healthy but `max(import_date)` frozen at 2026-07-10 (11 days stale), 0 imports in 7
+days, 3,322 records imported-but-not-on-web, the `testdatadb` `:3000` dashboard wedged, and both scheduled
+tasks erroring. Traced the outbound chain (DOS stations -> D2TESTNAS -> AD2 -> DB -> Hoffman API). Stations
+were still producing `.DAT` (live files dated that morning), so the break was the AD2<->NAS rsync pull: the
+sync log showed `rsync ... timeout waiting for daemon connection (code 35)` from ~2026-07-12.
+
+Root cause (transport): the NAS rsync daemon does a reverse-DNS lookup per connection; the NAS `resolv.conf`
+points at `nameserver 192.168.0.1`, which is **dead** (100% packet loss). Mike corrected an early mistake —
+the UDM is **.254**, not .1 — which pinned it: the NAS was pointed at a non-existent DNS server, so every
+rsync connection ate a **10-second** reverse-DNS timeout (`getent hosts 192.168.0.6` = 10.03s), blowing past
+the per-station 15-30s pull timeouts. Fixed with `reverse lookup = no` in `/etc/rsyncd.conf` (pull 10s->0.3s),
+restarted the dashboard service, ran the sync to drain the backlog, and used `_bulk_push_all.js` to push
+pending records to the website.
+
+Verified recovery: `import_date` advanced to now, `test_date` to 2026-07-20, 354 recent records pushed to the
+website. Added `jlehman@dataforth.com` to `notify.js` + `run-pipeline.ps1` and confirmed Graph `sendMail` = 202
+with a live test to John. Then, per Mike, did the proper DNS fix (NAS -> .254) and fixed the daily-report task,
+which had been dead since 2026-05-12 on a PowerShell parse error caused by mojibake em-dashes (4 lines).
+
+This continued a long same-day session whose earlier half was the IX PHP-FPM/OPcache fleet remediation
+(documented separately in clients/internal-infrastructure/reports + memory).
+
+## Key Decisions
+
+- Diagnosed before acting (per the fresh "profile before hardware/action" lesson) — pulled live DB recency +
+  sync logs + NAS state before concluding, rather than assuming which "flow" John meant.
+- Used the GuruRMM agent for AD2 (reliable) and Tailscale+plink for the NAS (100.85.152.90, password auth,
+  key auth denied). Held the NAS DNS change until Mike's go since that box also routes Neptune Exchange.
+- Applied `reverse lookup = no` as the immediate pull fix, then the proper `nmcli` DNS -> .254 as the root fix
+  (leaving `reverse lookup = no` as a harmless backstop).
+- Deferred adding John to reports until the pipeline was healthy + Graph delivery confirmed (202), so his first
+  report is real.
+- Treated the ~3,000 "skipped/unrenderable" pending records as a separate pre-existing data-quality item, not
+  the outage.
+
+## Problems Encountered
+
+- **rsync code-35 timeouts / no pulls since 2026-07-10** — NAS reverse-DNS 10s timeout (dead nameserver .1).
+  Fixed: `reverse lookup = no` + daemon restart; then `nmcli` DNS -> .254.
+- **Dashboard/API :3000 wedged** while service showed Running — restarted `testdatadb`; :3000 serving again.
+- **Imports not advancing after pull fixed** — import.js scans HISTLOGS+Recovery+station LOGS in ONE
+  transaction (slow, commits at end); my several reaped RMM commands spawned competing import.js. Cleaned up
+  orphans (kept the service node), triggered the scheduled task, watched `import_date` advance via a poller.
+- **Bulk web-push skipped 3,034 records** — pre-existing unrenderable backlog; only 53 recent were pending;
+  354 recent pushed OK. Not the outage.
+- **Daily task dead since 2026-05-12** — PowerShell PARSE error from mojibake em-dashes (lines 75/130/135/160).
+  My own earlier `Get-Content -Raw` + `Set-Content -Encoding UTF8` edit likely worsened the corruption. Fixed
+  by ASCII-cleaning (`-replace '[^\x00-\x7F]','-'`) + parse-test (`PSParser::Tokenize` = 0 errors); confirmed a
+  fresh pipeline log + stages running. Logged as `--friction`.
+- **Repeated RMM command reaps** on long/detached ops (>timeout) losing buffered output — worked around with
+  detached `Start-Process` + file logs + background pollers; got clean output via the commands API.
+
+## Configuration Changes
+
+- **D2TESTNAS `/etc/rsyncd.conf`** — added `reverse lookup = no` (backup `.bak-<ts>`); `systemctl restart rsync`.
+- **D2TESTNAS DNS** — `nmcli con mod "Wired connection 1" ipv4.dns 192.168.0.254`, `ipv4.ignore-auto-dns yes`,
+  `nmcli dev reapply eth0`. resolv.conf now `nameserver 192.168.0.254`.
+- **AD2 `C:\Shares\testdatadb\database\notify.js`** — `const TO` now includes `jlehman@dataforth.com` (backup
+  `.bak-20260721`).
+- **AD2 `C:\ProgramData\dataforth-uploader\run-pipeline.ps1`** — added jlehman to `toRecipients`; ASCII-cleaned
+  4 mojibake lines (backups `.bak-20260721`, `.bak-encfix-20260721`).
+- **AD2** — restarted `testdatadb` service; drained backlog via sync + `_bulk_push_all.js`.
+
+## Credentials & Secrets
+
+- D2TESTNAS root SSH: vault `clients/dataforth/d2testnas.sops.yaml` (username/password). Tailscale 100.85.152.90.
+  Hostkey `SHA256:DMMw8IJ4Aus54EAQOfJysLZ3Mw63WJ5R44ETiLMYeUI`.
+- rsync daemon: user `rsync` (password in `Sync-FromNAS-rsync.ps1`, plaintext — audit F5 flags for rotation).
+- Graph email creds: `C:\ProgramData\dataforth-uploader\credentials.json` (`GRAPH_TENANT_ID/CLIENT_ID/CLIENT_SECRET`),
+  sender `sysadmin@dataforth.com`. sendMail verified 202.
+
+## Infrastructure & Servers
+
+- **D2TESTNAS** 192.168.0.9 / Tailscale 100.85.152.90 — Debian 13, Samba SMB1 + rsync daemon (873, module
+  `test` -> /data/test), static IPv4, DNS now .254. Also physically hosts Neptune Exchange + advertises the
+  172.16.0.0/22 Tailscale route. Disk /data 87% (441G/512G).
+- **AD2** 192.168.0.6 — Win Server 2019, GuruRMM agent `4b9fe588-2a86-4855-912e-d90cf5f5e3e4`. `testdatadb`
+  (server.js) + `postgresql-18`. Dashboard `http://192.168.0.6:3000/`. Tasks `Sync-FromNAS` (15 min) +
+  `DataforthTestDatasheetUploader` (daily 02:30).
+- **UDM** 192.168.0.254 (DNS + gateway). `.1` is dead/unused.
+
+## Commands & Outputs
+
+- `getent hosts 192.168.0.6` on NAS: 10.031s (before) -> 0.007s (after DNS fix).
+- rsync enum from AD2: 10s -> 0.3s after `reverse lookup = no`.
+- DB recency: `{li: 2026-07-10 -> 2026-07-21 (now), lt: 2026-07-09 -> 2026-07-20, tot: 475871 -> 476342}`.
+- Web push (recent, test_date>=2026-07-10): pushed_recent=354, pend_recent=53, max_pushed=2026-07-20.
+- `_bulk_push_all.js`: created=0 updated=121 unchanged=37 errors=19 skipped=3034 (old unrenderable backlog).
+- Graph `sendMail` HTTP: 202 (test to jlehman + mike).
+- run-pipeline.ps1 parse test: PARSE_ERRORS=0 after ASCII fix; fresh log `pipeline-2026-07-21_08-36-09.log`.
+
+## Pending / Incomplete Tasks
+
+- **~3,000 old unrenderable records** stuck unpushed (skipped) — data-quality cleanup (stale/missing specs,
+  audit F4). Separate from this outage.
+- **Station-side inbound spec distribution (#32489)** — NWTOC v5.0 doesn't push master `.DAT`; separate.
+- Optional: rotate + vault the plaintext rsync/NAS/Postgres passwords (audit F5).
+- Confirm John actually received the test email (delivery was 202/accepted).
+
+## Reference Information
+
+- Wiki: `wiki/projects/dataforth-dos.md`; audit `clients/dataforth/docs/audits/2026-07-01-test-data-chain-audit-AD2.md`.
+- Memory: `.claude/memory/project_dataforth_datasheet_pipeline_recovery.md`.
+- Dashboard: `http://192.168.0.6:3000/`. Report emails: `[TestDataDB] Daily pipeline <PASS/FAIL> - <date>` from
+  sysadmin@dataforth.com. Logs: `C:\ProgramData\dataforth-uploader\logs\` + `upload-logs\`.