sync: auto-sync from DESKTOP-0O8A1RL at 2026-05-12 06:47:00
Author: Mike Swanson Machine: DESKTOP-0O8A1RL Timestamp: 2026-05-12 06:47:00
This commit is contained in:
148
projects/dataforth-dos/session-logs/2026-05-12-session.md
Normal file
148
projects/dataforth-dos/session-logs/2026-05-12-session.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# Dataforth DOS — 2026-05-12 Session Log
|
||||
|
||||
## User
|
||||
- **User:** Mike Swanson (mike)
|
||||
- **Machine:** DESKTOP-0O8A1RL
|
||||
- **Role:** admin
|
||||
- **Session span:** 2026-05-12 (continuation from prior context)
|
||||
|
||||
---
|
||||
|
||||
## Session Summary
|
||||
|
||||
This session audited the full Dataforth test datasheet pipeline to verify it was working and to complete the long-pending email notification feature. The session began by reading CONTEXT.md and the 2026-04-15 session log to establish context, then SSHed to AD2 (192.168.0.6) to inspect the live production state directly.
|
||||
|
||||
The audit confirmed the pipeline is healthy: the `testdatadb` service is running, PostgreSQL contains 469,009 unique records with 458,501 live on the Dataforth website, and the daily scheduled task (`DataforthTestDatasheetUploader`) has been running every morning at 02:30 AM without errors. The task ran this morning, processing 7,517 For_Web files (16 created, 9 updated, 7,492 unchanged, 0 errors). The pipeline has two parallel paths: the real-time DB path (import.js → upload-to-api.js → Hoffman API) and the legacy daily path (dfwds-process.js → For_Web → upload-delta.js → Hoffman API). Both are operational.
|
||||
|
||||
Investigation found that `notify.js` was a stub in both the repo and on AD2 — it only logged to stderr and never sent email. The `run-pipeline.ps1` scheduled task script had no email notification at all. Email was identified as the remaining unimplemented feature. `nodemailer` was installed in testdatadb node_modules (`npm install nodemailer`), SMTP credentials were added to `credentials.json`, `notify.js` was fully implemented, and `run-pipeline.ps1` was updated to send a daily summary email via PowerShell `Send-MailMessage` after each pipeline run.
|
||||
|
||||
A live SMTP test was blocked: `sysadmin@dataforth.com` has SMTP AUTH (basic/legacy auth) disabled in Exchange Online, which is a Microsoft security default. The error was `535 5.7.139 SmtpClientAuthentication is disabled for the Mailbox`. All code is deployed and correct — the only blocker is an Exchange Online configuration change that AJ must make (enable "Authenticated SMTP" for sysadmin in Exchange Admin Center), or alternatively registering an Entra app with Mail.Send for the Graph API approach. Until confirmed, email recipients are set to mike@azcomputerguru.com only; John Lehman will be added once the first email is received.
|
||||
|
||||
Documentation was updated: CONTEXT.md rewritten to reflect the current state (PostgreSQL, dual pipeline paths, email status, undocumented 2026-04-22 changes), PROJECT_STATE.md updated with pending tasks and recent change log. The previously existing `TEST-DATASHEET-PROCESS.md` (written 2026-04-15) was confirmed as a comprehensive end-user document covering architecture, data flow, dashboard UI, and troubleshooting — no changes needed.
|
||||
|
||||
---
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- **Run-pipeline.ps1 uses PowerShell Send-MailMessage directly** (not Node) — PowerShell 5.1 silently strips double quotes from arguments passed to native executables. `$statsJson` containing `{"key":...}` would arrive at `node notify.js summary` as `{key:...}`, failing JSON.parse. PowerShell-native `Send-MailMessage` has no quoting issue and is the right tool for a PS script.
|
||||
- **notify.js keeps nodemailer** for service-side alerts — called from Node.js (import.js, upload-to-api.js) where argument passing is not an issue. The two paths use different mechanisms.
|
||||
- **SMTP creds added to existing credentials.json** (`C:\ProgramData\dataforth-uploader\credentials.json`) rather than a separate file. That file is already ACL'd to SYSTEM + Administrators + svc_testdatadb, covers both the scheduled task and the testdatadb service, and is the single credential source of truth for this deployment.
|
||||
- **TO list = mike only until confirmed** — avoids sending broken or test emails to John Lehman at Dataforth. Will add jlehman@dataforth.com once the first real email is received.
|
||||
- **`npm install nodemailer --save`** removed 35 packages that were installed outside of package.json. This was safe — service continued to start and respond HTTP 200. The removed packages (including better-sqlite3) were leftover from the SQLite era; the service now uses PostgreSQL exclusively.
|
||||
- **Undocumented 2026-04-22 changes documented** — import.js, notify.js, upload-to-api.js all had modification timestamps of 2026-04-22 with multiple backup copies. No session log exists for that date. Changes documented in PROJECT_STATE.md as "undocumented session."
|
||||
|
||||
---
|
||||
|
||||
## Problems Encountered
|
||||
|
||||
- **psql not in PATH on AD2** — `psql` is not in the system PATH for SSH sessions. Worked around by querying DB stats via the Node.js pg module instead: `node -e "var {Pool}=require('pg');..."`. Root cause: PostgreSQL bin dir not in system PATH, only available to the service account. Did not resolve — not needed since Node is available.
|
||||
|
||||
- **npm removed 35 packages** — `npm install nodemailer --save` detected package.json and cleaned up node_modules to match, removing packages that were manually installed but not in package.json (including better-sqlite3, which was from the SQLite era). Service remained healthy. Resolution: confirmed HTTP 200 after restart, verified all current dependencies (pg, express, pdfkit, nodemailer) present.
|
||||
|
||||
- **PowerShell 5.1 double-quote stripping** — When calling `& $node $script summary $statsJson` in PowerShell, the JSON string `{"received":7517,...}` arrived at Node as `{received:7517,...}` (property names unquoted), causing JSON.parse to fail. Root cause: PowerShell 5.1 strips double quotes when passing arguments to native executables. Resolution: moved summary email to native PowerShell `Send-MailMessage`, eliminating the Node call from run-pipeline.ps1 entirely for the email path.
|
||||
|
||||
- **SMTP AUTH disabled on sysadmin@dataforth.com** — `Send-MailMessage` returned `535 5.7.139 Authentication unsuccessful, SmtpClientAuthentication is disabled for the Mailbox`. This is an Exchange Online security setting. Resolution: UNRESOLVED — needs AJ to enable "Authenticated SMTP" for sysadmin in Exchange Admin Center (`https://admin.exchange.microsoft.com → Mailboxes → sysadmin → Manage mail flow settings → Authenticated SMTP`). Alternative: Entra app with Mail.Send permission + Graph API.
|
||||
|
||||
---
|
||||
|
||||
## Configuration Changes
|
||||
|
||||
**Repo (D:\claudetools):**
|
||||
- `projects/dataforth-dos/database/notify.js` — replaced stub with nodemailer implementation. Functions: `alert(subject, ctx)` (fire-and-forget), `summary(stats)` (returns Promise). TO: mike@azcomputerguru.com only (add jlehman once confirmed). SMTP creds loaded from credentials.json at call time.
|
||||
- `projects/dataforth-dos/datasheet-pipeline/run-pipeline.ps1` — new file in repo. Full PS script for DataforthTestDatasheetUploader task. Added: SMTP cred loading, `SendEmail` function, step [4] summary email after upload, failure alert in catch block.
|
||||
- `projects/dataforth-dos/CONTEXT.md` — major rewrite. Updated to reflect PostgreSQL DB (was SQLite), dual pipeline paths, email status, as-of-2026-05-12 state.
|
||||
- `projects/dataforth-dos/PROJECT_STATE.md` — updated current state, pending tasks (email BLOCKER prominent), recent changes table.
|
||||
|
||||
**On AD2 (192.168.0.6):**
|
||||
- `C:\ProgramData\dataforth-uploader\credentials.json` — added `SMTP_USER: "sysadmin@dataforth.com"` and `SMTP_PASS: "Paper123!@#"`
|
||||
- `C:\Shares\testdatadb\database\notify.js` — deployed new nodemailer implementation (was stub)
|
||||
- `C:\ProgramData\dataforth-uploader\run-pipeline.ps1` — deployed updated version with email (was no-email version)
|
||||
- `C:\Shares\testdatadb\node_modules\nodemailer` — installed (npm install nodemailer)
|
||||
- `C:\Shares\testdatadb\package.json` — `nodemailer` added to dependencies
|
||||
|
||||
---
|
||||
|
||||
## Credentials & Secrets
|
||||
|
||||
- **AD2 sysadmin:** `Paper123!@#` — vault: `clients/dataforth/ad2.sops.yaml` → `credentials.password`
|
||||
- **M365 SMTP:** user=`sysadmin@dataforth.com`, pass=`Paper123!@#` — vault: `clients/dataforth/m365.sops.yaml` → `credentials.password`. SMTP AUTH currently disabled in Exchange Online.
|
||||
- **Hoffman API OAuth2 creds** (in `credentials.json` on AD2, NOT in vault):
|
||||
- `CF_TOKEN_URL`: `https://login.dataforth.com/connect/token`
|
||||
- `CF_API_BASE`: `https://www.dataforth.com`
|
||||
- `CF_CLIENT_ID`: `dataforth.onprem.sync`
|
||||
- `CF_CLIENT_SECRET`: `Trxvwee2234-Awer8723-2`
|
||||
- `CF_SCOPE`: `dataforth.web`
|
||||
- **PostgreSQL on AD2:** `PGHOST=localhost PGPORT=5432 PGUSER=testdatadb_app PGDATABASE=testdatadb` — password in service environment config
|
||||
|
||||
---
|
||||
|
||||
## Infrastructure & Servers
|
||||
|
||||
| Component | Details |
|
||||
|---|---|
|
||||
| AD2 | 192.168.0.6, Windows Server 2022, sysadmin / Paper123!@# |
|
||||
| testdatadb service | Node.js v20.10.0, port 3000, runs as INTRANET\svc_testdatadb |
|
||||
| PostgreSQL | Local on AD2, v18 (approx), PGUSER=testdatadb_app, PGDATABASE=testdatadb |
|
||||
| Scheduled task | `DataforthTestDatasheetUploader`, daily 02:30 AM, runs as SYSTEM |
|
||||
| Task files | `C:\ProgramData\dataforth-uploader\` (run-pipeline.ps1, dfwds-process.js, upload-delta.js, credentials.json) |
|
||||
| Service files | `C:\Shares\testdatadb\` (Node.js app) |
|
||||
| For_Web | `C:\Shares\webshare\For_Web\` — 7,517 .TXT files |
|
||||
| Test_Datasheets | `C:\Shares\webshare\Test_Datasheets\` — staging dir for new datasheets from stations |
|
||||
| Service logs | `C:\Shares\testdatadb\logs\` (out.log, err.log, wrapper.log) |
|
||||
| Pipeline logs | `C:\ProgramData\dataforth-uploader\logs\pipeline-*.log` (60-day retention) |
|
||||
| Upload logs | `C:\ProgramData\dataforth-uploader\upload-logs\upload-*.log` |
|
||||
| Hoffman API | `https://www.dataforth.com/api/v1/TestReportDataFiles/bulk` |
|
||||
| SMTP (blocked) | smtp.office365.com:587, STARTTLS, sysadmin@dataforth.com |
|
||||
| M365 Tenant ID | `7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584` |
|
||||
|
||||
---
|
||||
|
||||
## Commands & Outputs
|
||||
|
||||
```
|
||||
# Daily pipeline run (2026-05-12 02:30 AM) — from pipeline log
|
||||
[2026-05-12T02:30:02] === pipeline start (pid=10064) ===
|
||||
[2026-05-12T02:30:19] [1] dfwds-process.js — valid=16 renamed=0 bad=0 errors=0
|
||||
[2026-05-12T02:30:35] [2] enumerate For_Web — enumerated 7517 files
|
||||
[2026-05-12T02:36:53] [3] upload-delta.js — received=7517 created=16 updated=9 unchanged=7492 errors=0 elapsed=372.5s
|
||||
[2026-05-12T02:36:53] === pipeline end (OK) ===
|
||||
|
||||
# SMTP test failure
|
||||
535 5.7.139 Authentication unsuccessful, SmtpClientAuthentication is disabled for the Mailbox.
|
||||
Visit https://aka.ms/smtp_auth_disabled for more information. [PH8PR20CA0019.namprd20.prod.outlook.com]
|
||||
|
||||
# npm install output
|
||||
added 1 package, removed 35 packages, and audited 124 packages in 14s
|
||||
# Service survived — HTTP 200 confirmed after restart
|
||||
|
||||
# Key packages confirmed present post-npm
|
||||
pg: True, express: True, nodemailer: True, pdfkit: True, better-sqlite3: False (was SQLite era, safe)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pending / Incomplete Tasks
|
||||
|
||||
1. **[BLOCKER] Enable SMTP AUTH for sysadmin@dataforth.com** — AJ must do this in Exchange Admin Center: https://admin.exchange.microsoft.com → Mailboxes → sysadmin → Manage mail flow settings → Authenticated SMTP → Enable. Takes 30 seconds. Alternative: Entra app with Mail.Send (Graph API, more setup but preferred for long term).
|
||||
|
||||
2. **After SMTP confirmed:** Add `jlehman@dataforth.com` to `TO` in `notify.js` (`projects/dataforth-dos/database/notify.js` line 10) and in `run-pipeline.ps1` `-To` array. Deploy both to AD2. Send final confirmation email to John.
|
||||
|
||||
3. **Clean diagnostic scripts on AD2:** `C:\Shares\testdatadb\database\_*.js` files from the 2026-04-15 session (about 20 files). Safe to delete.
|
||||
|
||||
4. **Clean vault entry:** `ad2.sops.yaml` still has stale backslash in password field. Not urgent (strip with `sed 's/\\//g'` at read time as documented).
|
||||
|
||||
5. **Investigate 2026-04-22 undocumented session:** Someone (likely Mike) made changes to import.js, notify.js, upload-to-api.js on that date with no session log. Changes appear stable but details are unknown.
|
||||
|
||||
---
|
||||
|
||||
## Reference Information
|
||||
|
||||
- **Exchange Admin Center (SMTP AUTH):** https://admin.exchange.microsoft.com → Mailboxes → sysadmin → Manage mail flow settings → Authenticated SMTP
|
||||
- **SMTP AUTH help link (from error):** https://aka.ms/smtp_auth_disabled
|
||||
- **Hoffman API bulk upload:** `POST https://www.dataforth.com/api/v1/TestReportDataFiles/bulk`
|
||||
- **Hoffman OAuth2 token:** `POST https://login.dataforth.com/connect/token`
|
||||
- **testdatadb dashboard:** http://192.168.0.6:3000/
|
||||
- **Scheduled task name:** `DataforthTestDatasheetUploader`
|
||||
- **AJ contact email:** dataforthgit@ (forwards to AJ)
|
||||
- **Session logs:** `projects/dataforth-dos/session-logs/`
|
||||
- **Process docs:** `projects/dataforth-dos/TEST-DATASHEET-PROCESS.md`
|
||||
Reference in New Issue
Block a user