From ada02831e3a3bb1d3cabcfd344f4763a3db79d0c Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Tue, 21 Jul 2026 08:43:38 -0700 Subject: [PATCH] sync: auto-sync from GURU-5070 at 2026-07-21 08:42:56 Author: Mike Swanson Machine: GURU-5070 Timestamp: 2026-07-21 08:42:56 --- .claude/memory/MEMORY.md | 1 + ...t_dataforth_datasheet_pipeline_recovery.md | 37 ++++++ ...mike-datasheet-pipeline-outage-recovery.md | 121 ++++++++++++++++++ errorlog.md | Bin 155770 -> 156418 bytes 4 files changed, 159 insertions(+) create mode 100644 .claude/memory/project_dataforth_datasheet_pipeline_recovery.md create mode 100644 clients/dataforth/session-logs/2026-07/2026-07-21-mike-datasheet-pipeline-outage-recovery.md diff --git a/.claude/memory/MEMORY.md b/.claude/memory/MEMORY.md index 6ce4adbd..e5cda106 100644 --- a/.claude/memory/MEMORY.md +++ b/.claude/memory/MEMORY.md @@ -27,6 +27,7 @@ - [Radio Show Website](reference_radio_website.md) — Astro static site at radio.azcomputerguru.com on IX server. - [IX PHP-FPM/OPcache remediation](project_ix_php_fpm_opcache_remediation.md) — IX server default is suPHP/CGI per-request with OPcache cold: 71/72 sites recompile PHP every request (1-5.6s TTFB). Fix = per-vhost PHP-FPM. ezfast pilot done (3.1s->0.44s); fleet rollout proposed. - [Profile before hardware](feedback_profile_before_hardware.md) — For ANY performance complaint, measure/decompose the actual bottleneck BEFORE proposing hardware; config/cheapest cause first. (IX: $600 SSDs recommended before finding the real cause was OPcache-off — a config issue.) +- [Dataforth datasheet pipeline recovery](project_dataforth_datasheet_pipeline_recovery.md) — 2026-07-21 recovery of DOS datasheet pipeline: NAS (D2TESTNAS, TS 100.85.152.90) DNS pointed at dead .1 (UDM is .254) → rsync reverse-DNS 10s timeouts stalled pulls; fixed. Daily task broke on PS mojibake parse error. import.js/test_date/_bulk_push_all internals. jlehman added to reports. - [IX Server Access](reference_ix_server_access.md) — `ix.azcomputerguru.com` / 172.16.3.10. Reachable when Tailscale is on (no VPN). SSH currently uses sshpass with root password; key auth from GURU-5070 not configured yet (was CachyOS, now Win11 — verify). - [Cloudflare access](reference_cloudflare_access.md) — Cloudflare API creds in SOPS `services/cloudflare.sops.yaml` (full DNS + account tokens; azcomputerguru zone_id 1beb9917...). azcomputerguru.com DNS is on Cloudflare (not IX) — edit via Cloudflare API, not whmapi1. - [Matomo Analytics](reference_matomo_analytics.md) — Self-hosted analytics at analytics.azcomputerguru.com, site IDs, tracking for all 3 sites. diff --git a/.claude/memory/project_dataforth_datasheet_pipeline_recovery.md b/.claude/memory/project_dataforth_datasheet_pipeline_recovery.md new file mode 100644 index 00000000..53203be0 --- /dev/null +++ b/.claude/memory/project_dataforth_datasheet_pipeline_recovery.md @@ -0,0 +1,37 @@ +--- +name: project_dataforth_datasheet_pipeline_recovery +description: Dataforth DOS datasheet pipeline "no new datasheets" recovery (2026-07-21) — NAS rsync reverse-DNS timeout + daily-task parse error; fixes + gotchas. +metadata: + type: project +--- + +Dataforth test-datasheet pipeline stopped producing new datasheets ~2026-07-10. Full recovery 2026-07-21. + +**Root cause (transport):** D2TESTNAS (NAS, 192.168.0.9; Tailscale **100.85.152.90**, root+password, key auth +denied; hostkey SHA256:DMMw8IJ4Aus54EAQOfJysLZ3Mw63WJ5R44ETiLMYeUI) had `resolv.conf nameserver 192.168.0.1` +which is **DEAD** (100% packet loss). The **UDM is .254** (not .1). rsyncd does a reverse-DNS lookup per +connection by default → every rsync connection ate a **10s DNS timeout** (`getent hosts` = 10.03s), so the +per-station `.DAT` pulls (15-30s timeouts) failed "code 35" and no test data reached AD2. Fixes: (1) `reverse +lookup = no` in `/etc/rsyncd.conf` + `systemctl restart rsync` (backstop; pull 10s->0.3s); (2) proper: +`nmcli con mod "Wired connection 1" ipv4.dns 192.168.0.254; nmcli con mod ... ipv4.ignore-auto-dns yes; nmcli +dev reapply eth0` (reverse lookup now 0.007s). NAS is Debian 13, static IPv4. + +**Second issue (daily report):** `DataforthTestDatasheetUploader` task (run-pipeline.ps1) failed since May 12 — +a **PowerShell PARSE error from mojibake em-dashes** (lines 75/130/135/160 had corrupted `—`). Fix: ASCII-clean +those lines (`-replace '[^\x00-\x7F]','-'`), verify with `[System.Management.Automation.PSParser]::Tokenize`. +**GOTCHA: editing a .ps1 containing non-ASCII via `Get-Content -Raw` + `Set-Content -Encoding UTF8` double- +encodes/corrupts the non-ASCII** — keep pipeline scripts ASCII-only; parse-test after any edit. See +[[feedback_windows_quote_stripping]]. + +**Pipeline internals learned:** import.js (`C:\Shares\testdatadb\database`) scans HISTLOGS + Recovery-TEST + +station LOGS (`C:/Shares/test/TS-*/LOGS`) in **one transaction** — slow, commits only at the end; the sync +(`Sync-FromNAS-rsync.ps1`, rsync daemon user `rsync`) pulls `.DAT` then calls it. `test_date` is a **TEXT** +column (compare as ISO text, no `date` cast; `to_char` fails). `_bulk_push_all.js` pushes pending PASS to the +Hoffman API (idempotent). Dashboard/API `:3000` (restart `testdatadb` service if wedged). Report recipients: +`notify.js` `const TO` + `run-pipeline.ps1` `toRecipients` — **jlehman@dataforth.com added** 2026-07-21; Graph +sendMail confirmed **202** (creds in `C:\ProgramData\dataforth-uploader\credentials.json` GRAPH_*). + +**Still open:** ~3,000 old **unrenderable** records (skipped on push — stale/missing specs, audit F4); the +station-side inbound spec distribution (#32489) is separate. AD2 reached via GuruRMM agent +(4b9fe588-...), not directly (192.168.0.x not on the Tailscale 172.16.0.0/22 route). Report: +`clients/internal-infrastructure`? no — see clients/dataforth session log 2026-07-21. See [[project_ix_php_fpm_opcache_remediation]] for the same-session IX work. diff --git a/clients/dataforth/session-logs/2026-07/2026-07-21-mike-datasheet-pipeline-outage-recovery.md b/clients/dataforth/session-logs/2026-07/2026-07-21-mike-datasheet-pipeline-outage-recovery.md new file mode 100644 index 00000000..b8957cef --- /dev/null +++ b/clients/dataforth/session-logs/2026-07/2026-07-21-mike-datasheet-pipeline-outage-recovery.md @@ -0,0 +1,121 @@ +## User +- **User:** Mike Swanson (mike) +- **Machine:** GURU-5070 +- **Role:** admin + +# Dataforth DOS Test-Datasheet Pipeline — Outage Recovery (John Lehman) + +**Date:** 2026-07-21 · **Client:** Dataforth Corporation · **Systems:** D2TESTNAS (NAS), AD2 (pipeline host) + +## Session Summary + +John Lehman reported the "DOS test flow isn't working" (no new datasheets on dataforth.com), asked to be +included on the Success/Failure reports, and asked where to find a report. Loaded the Dataforth DOS project +context (wiki + the 2026-07-01 ground-truth audit), then diagnosed live via the GuruRMM agent on AD2 (the box +isn't directly routable — 192.168.0.x is not on the Tailscale 172.16.0.0/22 route). + +Ground truth: PostgreSQL healthy but `max(import_date)` frozen at 2026-07-10 (11 days stale), 0 imports in 7 +days, 3,322 records imported-but-not-on-web, the `testdatadb` `:3000` dashboard wedged, and both scheduled +tasks erroring. Traced the outbound chain (DOS stations -> D2TESTNAS -> AD2 -> DB -> Hoffman API). Stations +were still producing `.DAT` (live files dated that morning), so the break was the AD2<->NAS rsync pull: the +sync log showed `rsync ... timeout waiting for daemon connection (code 35)` from ~2026-07-12. + +Root cause (transport): the NAS rsync daemon does a reverse-DNS lookup per connection; the NAS `resolv.conf` +points at `nameserver 192.168.0.1`, which is **dead** (100% packet loss). Mike corrected an early mistake — +the UDM is **.254**, not .1 — which pinned it: the NAS was pointed at a non-existent DNS server, so every +rsync connection ate a **10-second** reverse-DNS timeout (`getent hosts 192.168.0.6` = 10.03s), blowing past +the per-station 15-30s pull timeouts. Fixed with `reverse lookup = no` in `/etc/rsyncd.conf` (pull 10s->0.3s), +restarted the dashboard service, ran the sync to drain the backlog, and used `_bulk_push_all.js` to push +pending records to the website. + +Verified recovery: `import_date` advanced to now, `test_date` to 2026-07-20, 354 recent records pushed to the +website. Added `jlehman@dataforth.com` to `notify.js` + `run-pipeline.ps1` and confirmed Graph `sendMail` = 202 +with a live test to John. Then, per Mike, did the proper DNS fix (NAS -> .254) and fixed the daily-report task, +which had been dead since 2026-05-12 on a PowerShell parse error caused by mojibake em-dashes (4 lines). + +This continued a long same-day session whose earlier half was the IX PHP-FPM/OPcache fleet remediation +(documented separately in clients/internal-infrastructure/reports + memory). + +## Key Decisions + +- Diagnosed before acting (per the fresh "profile before hardware/action" lesson) — pulled live DB recency + + sync logs + NAS state before concluding, rather than assuming which "flow" John meant. +- Used the GuruRMM agent for AD2 (reliable) and Tailscale+plink for the NAS (100.85.152.90, password auth, + key auth denied). Held the NAS DNS change until Mike's go since that box also routes Neptune Exchange. +- Applied `reverse lookup = no` as the immediate pull fix, then the proper `nmcli` DNS -> .254 as the root fix + (leaving `reverse lookup = no` as a harmless backstop). +- Deferred adding John to reports until the pipeline was healthy + Graph delivery confirmed (202), so his first + report is real. +- Treated the ~3,000 "skipped/unrenderable" pending records as a separate pre-existing data-quality item, not + the outage. + +## Problems Encountered + +- **rsync code-35 timeouts / no pulls since 2026-07-10** — NAS reverse-DNS 10s timeout (dead nameserver .1). + Fixed: `reverse lookup = no` + daemon restart; then `nmcli` DNS -> .254. +- **Dashboard/API :3000 wedged** while service showed Running — restarted `testdatadb`; :3000 serving again. +- **Imports not advancing after pull fixed** — import.js scans HISTLOGS+Recovery+station LOGS in ONE + transaction (slow, commits at end); my several reaped RMM commands spawned competing import.js. Cleaned up + orphans (kept the service node), triggered the scheduled task, watched `import_date` advance via a poller. +- **Bulk web-push skipped 3,034 records** — pre-existing unrenderable backlog; only 53 recent were pending; + 354 recent pushed OK. Not the outage. +- **Daily task dead since 2026-05-12** — PowerShell PARSE error from mojibake em-dashes (lines 75/130/135/160). + My own earlier `Get-Content -Raw` + `Set-Content -Encoding UTF8` edit likely worsened the corruption. Fixed + by ASCII-cleaning (`-replace '[^\x00-\x7F]','-'`) + parse-test (`PSParser::Tokenize` = 0 errors); confirmed a + fresh pipeline log + stages running. Logged as `--friction`. +- **Repeated RMM command reaps** on long/detached ops (>timeout) losing buffered output — worked around with + detached `Start-Process` + file logs + background pollers; got clean output via the commands API. + +## Configuration Changes + +- **D2TESTNAS `/etc/rsyncd.conf`** — added `reverse lookup = no` (backup `.bak-`); `systemctl restart rsync`. +- **D2TESTNAS DNS** — `nmcli con mod "Wired connection 1" ipv4.dns 192.168.0.254`, `ipv4.ignore-auto-dns yes`, + `nmcli dev reapply eth0`. resolv.conf now `nameserver 192.168.0.254`. +- **AD2 `C:\Shares\testdatadb\database\notify.js`** — `const TO` now includes `jlehman@dataforth.com` (backup + `.bak-20260721`). +- **AD2 `C:\ProgramData\dataforth-uploader\run-pipeline.ps1`** — added jlehman to `toRecipients`; ASCII-cleaned + 4 mojibake lines (backups `.bak-20260721`, `.bak-encfix-20260721`). +- **AD2** — restarted `testdatadb` service; drained backlog via sync + `_bulk_push_all.js`. + +## Credentials & Secrets + +- D2TESTNAS root SSH: vault `clients/dataforth/d2testnas.sops.yaml` (username/password). Tailscale 100.85.152.90. + Hostkey `SHA256:DMMw8IJ4Aus54EAQOfJysLZ3Mw63WJ5R44ETiLMYeUI`. +- rsync daemon: user `rsync` (password in `Sync-FromNAS-rsync.ps1`, plaintext — audit F5 flags for rotation). +- Graph email creds: `C:\ProgramData\dataforth-uploader\credentials.json` (`GRAPH_TENANT_ID/CLIENT_ID/CLIENT_SECRET`), + sender `sysadmin@dataforth.com`. sendMail verified 202. + +## Infrastructure & Servers + +- **D2TESTNAS** 192.168.0.9 / Tailscale 100.85.152.90 — Debian 13, Samba SMB1 + rsync daemon (873, module + `test` -> /data/test), static IPv4, DNS now .254. Also physically hosts Neptune Exchange + advertises the + 172.16.0.0/22 Tailscale route. Disk /data 87% (441G/512G). +- **AD2** 192.168.0.6 — Win Server 2019, GuruRMM agent `4b9fe588-2a86-4855-912e-d90cf5f5e3e4`. `testdatadb` + (server.js) + `postgresql-18`. Dashboard `http://192.168.0.6:3000/`. Tasks `Sync-FromNAS` (15 min) + + `DataforthTestDatasheetUploader` (daily 02:30). +- **UDM** 192.168.0.254 (DNS + gateway). `.1` is dead/unused. + +## Commands & Outputs + +- `getent hosts 192.168.0.6` on NAS: 10.031s (before) -> 0.007s (after DNS fix). +- rsync enum from AD2: 10s -> 0.3s after `reverse lookup = no`. +- DB recency: `{li: 2026-07-10 -> 2026-07-21 (now), lt: 2026-07-09 -> 2026-07-20, tot: 475871 -> 476342}`. +- Web push (recent, test_date>=2026-07-10): pushed_recent=354, pend_recent=53, max_pushed=2026-07-20. +- `_bulk_push_all.js`: created=0 updated=121 unchanged=37 errors=19 skipped=3034 (old unrenderable backlog). +- Graph `sendMail` HTTP: 202 (test to jlehman + mike). +- run-pipeline.ps1 parse test: PARSE_ERRORS=0 after ASCII fix; fresh log `pipeline-2026-07-21_08-36-09.log`. + +## Pending / Incomplete Tasks + +- **~3,000 old unrenderable records** stuck unpushed (skipped) — data-quality cleanup (stale/missing specs, + audit F4). Separate from this outage. +- **Station-side inbound spec distribution (#32489)** — NWTOC v5.0 doesn't push master `.DAT`; separate. +- Optional: rotate + vault the plaintext rsync/NAS/Postgres passwords (audit F5). +- Confirm John actually received the test email (delivery was 202/accepted). + +## Reference Information + +- Wiki: `wiki/projects/dataforth-dos.md`; audit `clients/dataforth/docs/audits/2026-07-01-test-data-chain-audit-AD2.md`. +- Memory: `.claude/memory/project_dataforth_datasheet_pipeline_recovery.md`. +- Dashboard: `http://192.168.0.6:3000/`. Report emails: `[TestDataDB] Daily pipeline - ` from + sysadmin@dataforth.com. Logs: `C:\ProgramData\dataforth-uploader\logs\` + `upload-logs\`. diff --git a/errorlog.md b/errorlog.md index 4b5a920cd44336566470d34cc6f6d759d9099d00..77951072c47f89e01c9125b465dfba5132b91067 100644 GIT binary patch delta 628 zcmYk4!EO^V7=!`k(o-J*4kK}BBIFH;0F^{ViW)!_qKep5IfN3q_P<%DyY|XBrvJNn>lvvRyewQh9vDG_y*0W%N0s zqVyabN#!~aMtiPwRJ74-kRG3$ph+c@+~wq2SSgOzX6LnV$+B7fyDH~9kx9O*Yd9;TX=FF%~3jv18lE1HkYK;HMPJqPd1>N zx{3r-;jenIlpR&pEJ*m7`+&F(U7d{XK3+e#*5wOc6eMvvNTGR2Az83O=Xp2jJUdJtCmkGwl5jSBd(e+7f&~|T z8IEmkVcn-s5{v^FjVw8{C{{^$$qbcIIy5~zO~E~aUvq7Zp0_1&*}^(kB9ESNNNBsL rL2HbM;7dpS@Ez)~wZ2QY!(*Bx48HlbfB*BO@nzDu`)$&=_wfB6gRI!z delta 22 ecmZoV#`)_2=Z0{W=KE~z_t_Y?-)Cb|jsXC5LJ1N8