From b1a588d0db13ec656033ba5b30d6d4efbee810b8 Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Tue, 12 May 2026 06:47:00 -0700 Subject: [PATCH] sync: auto-sync from DESKTOP-0O8A1RL at 2026-05-12 06:47:00 Author: Mike Swanson Machine: DESKTOP-0O8A1RL Timestamp: 2026-05-12 06:47:00 --- projects/dataforth-dos/CONTEXT.md | 45 ++-- projects/dataforth-dos/PROJECT_STATE.md | 17 +- projects/dataforth-dos/database/notify.js | 203 ++++++++++++++---- .../datasheet-pipeline/run-pipeline.ps1 | 145 +++++++++++++ .../session-logs/2026-05-12-session.md | 148 +++++++++++++ projects/msp-tools/guru-rmm | 2 +- 6 files changed, 488 insertions(+), 72 deletions(-) create mode 100644 projects/dataforth-dos/datasheet-pipeline/run-pipeline.ps1 create mode 100644 projects/dataforth-dos/session-logs/2026-05-12-session.md diff --git a/projects/dataforth-dos/CONTEXT.md b/projects/dataforth-dos/CONTEXT.md index 7b27d20..d418377 100644 --- a/projects/dataforth-dos/CONTEXT.md +++ b/projects/dataforth-dos/CONTEXT.md @@ -1,7 +1,7 @@ # Dataforth DOS Project - Context -**Last Updated:** 2026-04-14 -**Status:** Active - Datasheet Pipeline Extended for SCMVAS/SCMHVAS +**Last Updated:** 2026-05-12 +**Status:** Active - Email notifications implemented (pending M365 SMTP AUTH config) ## Quick Start - Infrastructure Overview @@ -25,26 +25,35 @@ bash D:/vault/scripts/vault.sh get-field clients/dataforth/ad1.sops.yaml credent ## Current State (READ THIS FIRST) -### Recent Work (2026-04-11/12) -**Extended Test Datasheet Pipeline for SCMVAS-Mxxx and SCMHVAS-Mxxxx families** -- Added VASLOG parser support (multiline CSV .DAT format) -- Created accuracy-only datasheet template (simple format, no hvin.dat lookup) -- Implemented pass-through for Engineering-Tested .txt files -- **Backfilled 27,503 historical records** (438 required regex patch for QB STR$() format quirk) -- **434 Engineering .txt files** imported and published -- Deployed to AD2, service restarted, web publishing verified +### As of 2026-05-12 -**Status:** ✅ Complete, production-deployed +**Production pipeline is healthy and fully operational.** Daily task runs at 02:30 AM, no errors. testdatadb service running on AD2. -**Critical Files Changed:** 5 modified, 1 new parser -- server/parsers/vaslog.js (new) -- server/templates/datasheet-exact.js (SCMVAS/SCMHVAS branch added) -- server/database/import.js (recursive flag fix, VASLOG_ENG support) -- server/parsers/spec-reader.js (stub for SCMVAS/SCMHVAS) -- deploy/deploy-to-ad2.py (vault-based credentials) +**Email notification status:** Code is deployed and wired. Blocked by M365 config — `sysadmin@dataforth.com` has SMTP AUTH (basic auth) disabled in Exchange Online. **Action required from AJ:** Enable "Authenticated SMTP" for sysadmin in Exchange Admin Center, OR create an Entra app with Mail.Send permission for Graph API approach. Recipients once confirmed: jlehman@dataforth.com + mike@azcomputerguru.com. Currently set to mike only for testing. + +**DB stats (as of 2026-04-15 release):** +- 469,009 unique SNs in PostgreSQL +- 458,501 live on Dataforth website +- 10,508 internal only (7,905 missing specs, 2,426 Hoffman errors, 177 FAIL) +- 7,517 For_Web files (legacy path, still used by scheduled task) + +### Pipeline Architecture (two parallel paths) + +1. **Real-time (testdatadb service):** .dat files → import.js → PostgreSQL → upload-to-api.js → Hoffman API. notify.alert() fires on errors. +2. **Daily scheduled task (02:30 AM):** dfwds-process.js moves Test_Datasheets → For_Web → upload-delta.js → Hoffman API. Sends summary email on completion. + +### Recent Work Summary + +**2026-04-15:** Major release — DB dedup (2.89M→469K rows), FAIL→PASS retest rule, For_Web filesystem dependency eliminated (upload-to-api.js now renders in memory), bulk push of 170,984 records to Hoffman, dashboard UI upgrades (pink tint, push buttons, bulk push, website status filter). + +**2026-04-22 (undocumented):** Modifications to import.js, notify.js, upload-to-api.js on AD2. No session log exists for this date. + +**2026-05-12:** Email notification implementation — nodemailer deployed, credentials.json updated with SMTP creds, run-pipeline.ps1 updated for daily summary email, TEST-DATASHEET-PROCESS.md created as full documentation. Blocked on M365 SMTP AUTH (see above). **Session Logs:** -- **2026-04-12-session.md** - Implementation, deploy, backfill, patch (DEFINITIVE) +- **2026-05-12-session.md** - Email implementation, pipeline audit (CURRENT) +- **2026-04-15-session.md** - Major release (DB dedup, push, UI) +- **2026-04-12-session.md** - SCMVAS/SCMHVAS implementation (DEFINITIVE) - **2026-04-11-discovery-session.md** - Discovery phase ### testdatadb Service (on AD2) diff --git a/projects/dataforth-dos/PROJECT_STATE.md b/projects/dataforth-dos/PROJECT_STATE.md index 4aae689..7099cc6 100644 --- a/projects/dataforth-dos/PROJECT_STATE.md +++ b/projects/dataforth-dos/PROJECT_STATE.md @@ -2,7 +2,7 @@ > READ THIS before starting work on this project. > UPDATE THIS when you begin work (claim a lock) and when you finish (release lock + log changes). -> Last updated: 2026-04-20 +> Last updated: 2026-05-12 --- @@ -19,11 +19,9 @@ ## Current State **Status:** ACTIVE / DEPLOYED -**Last Activity:** 2026-04-14 +**Last Activity:** 2026-05-12 -Legacy DOS hardware test pipeline for Dataforth Corporation. Node.js service (`testdatadb`) on AD2 (192.168.0.6) parses test log files from DOS-era test stations, stores 27k+ records in SQLite, and publishes datasheets to the web. As of 2026-04-12, pipeline was extended to support SCMVAS/SCMHVAS product families with a new VASLOG parser and accuracy-only datasheet template. 27,503 historical records backfilled; 434 Engineering-Tested .txt files imported. Service is production-deployed and running. - -Remaining work: deploy session manager to SAGE-SQL, verify historical backfill accuracy for edge cases, continue any new product family extensions. +Pipeline is healthy and fully operational. testdatadb service running on AD2 (PostgreSQL backend, 469K records, 458K on website). Daily scheduled task (`DataforthTestDatasheetUploader`) runs at 02:30 AM and ran clean this morning (16 created, 9 updated, 0 errors). Email notification code deployed — blocked on M365 SMTP AUTH configuration (AJ must enable "Authenticated SMTP" for sysadmin@dataforth.com in Exchange Admin Center, or create an Entra app with Mail.Send). See CONTEXT.md for details. --- @@ -54,10 +52,10 @@ bash D:/vault/scripts/vault.sh get-field clients/dataforth/ad1.sops.yaml credent ## Pending / Next Up -- [ ] Deploy session manager to SAGE-SQL -- [ ] SCMVAS/SCMHVAS pipeline further extension (if new families identified) -- [ ] Verify 27,503 historical records backfill accuracy (438 plain-decimal edge cases patched 2026-04-12) +- [ ] **BLOCKER (email):** AJ enable "Authenticated SMTP" for sysadmin@dataforth.com in Exchange Admin Center (https://admin.exchange.microsoft.com → Mailboxes → sysadmin → Manage mail flow settings → Authenticated SMTP). Once done, re-run test email to confirm, then add jlehman@dataforth.com to TO list in notify.js and run-pipeline.ps1. +- [ ] After email confirmed working: add John Lehman (jlehman@dataforth.com) to notify.js `TO` constant and run-pipeline.ps1 `SendEmail -To` list. Redeploy both. - [ ] Clean vault entry: ad2.sops.yaml has stale backslash in password (remove `\` escape) +- [ ] Clean up diagnostic scripts from C:\Shares\testdatadb\database\ (all underscore-prefixed files from 2026-04-15 session) --- @@ -65,6 +63,9 @@ bash D:/vault/scripts/vault.sh get-field clients/dataforth/ad1.sops.yaml credent | Date | By | Change | Status | |------|-----|--------|--------| +| 2026-05-12 | Mike | Email notifications: nodemailer deployed, credentials.json SMTP creds, run-pipeline.ps1 summary email, TEST-DATASHEET-PROCESS.md docs, CONTEXT.md updated | DEPLOYED (blocked on M365 SMTP AUTH) | +| 2026-04-22 | Mike | Modifications to import.js, notify.js, upload-to-api.js (undocumented session) | DEPLOYED | +| 2026-04-15 | Mike | DB dedup (2.89M→469K), FAIL→PASS rule, For_Web eliminated, 170,984 records pushed to Hoffman, dashboard UI | DEPLOYED | | 2026-04-12 | Mike | SCMVAS/SCMHVAS pipeline: new VASLOG parser, accuracy-only template, 27,503 records backfilled, 434 ENG txt files imported | DEPLOYED | | 2026-04-11 | Mike | Discovery session: explored VASLOG .DAT format, hvin.dat binary structure | RESEARCH | diff --git a/projects/dataforth-dos/database/notify.js b/projects/dataforth-dos/database/notify.js index 6d8a8ce..ca584cb 100644 --- a/projects/dataforth-dos/database/notify.js +++ b/projects/dataforth-dos/database/notify.js @@ -1,71 +1,184 @@ -/** - * Fire-and-forget alert notification module. - * Email sending is not yet implemented (Entra app pending). - * Logs a formatted preview of what would be emailed to stderr. - * Never throws — callers do not need to guard this. - */ +'use strict'; -const os = require('os'); +const os = require('os'); +const fs = require('fs'); +const path = require('path'); -const TO = 'mike@azcomputerguru.com'; -const FROM = 'sysadmin@dataforth.com'; -const HOST = os.hostname(); -const PREFIX = '[NOTIFY]'; -const SEP_EQ = '='.repeat(60); -const SEP_DAS = '-'.repeat(60); +const CREDS_PATH = 'C:\\ProgramData\\dataforth-uploader\\credentials.json'; +const TO = ['mike@azcomputerguru.com']; // add jlehman@dataforth.com once confirmed working +const FROM = 'sysadmin@dataforth.com'; +const HOST = os.hostname(); +const PREFIX = '[NOTIFY]'; +const SEP_EQ = '='.repeat(60); +const SEP_DAS = '-'.repeat(60); -/** - * Log a formatted email preview to stderr. - * - * @param {string} subject - Email subject line (prefixed with [TestDataDB] automatically) - * @param {object} [context={}] - * @param {string} [context.stage] - Pipeline stage where the alert fired - * @param {string} [context.error] - Error message or description - * @param {string[]} [context.details] - Additional detail lines - * @param {object} [context.stats] - Arbitrary stats object, serialized as JSON - */ -function alert(subject, context) { - context = context || {}; +function loadSmtpCreds() { + try { + const raw = fs.readFileSync(CREDS_PATH, 'utf8'); + const c = JSON.parse(raw); + if (!c.SMTP_USER || !c.SMTP_PASS) { + process.stderr.write(`${PREFIX} SMTP_USER/SMTP_PASS not present in credentials.json — skipping email\n`); + return null; + } + return { user: c.SMTP_USER, pass: c.SMTP_PASS }; + } catch (e) { + process.stderr.write(`${PREFIX} Could not load credentials.json: ${e.message} — skipping email\n`); + return null; + } +} + +function buildTransport(creds) { + const nodemailer = require('nodemailer'); + return nodemailer.createTransport({ + host: 'smtp.office365.com', + port: 587, + requireTLS: true, + auth: { + user: creds.user, + pass: creds.pass, + }, + }); +} + +async function sendMail(subject, text) { + const creds = loadSmtpCreds(); + if (!creds) return; + + try { + const transport = buildTransport(creds); + await transport.sendMail({ + from: FROM, + to: TO, + subject, + text, + }); + } catch (e) { + process.stderr.write(`${PREFIX} sendMail failed: ${e.message}\n`); + } +} + +function buildAlertText(subject, context) { const lines = []; + const ts = new Date().toISOString(); - lines.push(`${PREFIX} ${SEP_EQ}`); - lines.push(`${PREFIX} TO: ${TO}`); - lines.push(`${PREFIX} FROM: ${FROM} (pending Entra app)`); - lines.push(`${PREFIX} SUBJECT: [TestDataDB] ${subject}`); - lines.push(`${PREFIX} ${SEP_DAS}`); - lines.push(`${PREFIX} Host: ${HOST}`); - lines.push(`${PREFIX} Time: ${new Date().toISOString()}`); + lines.push(`Host: ${HOST}`); + lines.push(`Time: ${ts}`); + lines.push(`Subject: [TestDataDB] ALERT: ${subject}`); + lines.push(SEP_DAS); if (context.stage !== undefined) { - lines.push(`${PREFIX} Stage: ${context.stage}`); + lines.push(`Stage: ${context.stage}`); } if (context.error !== undefined) { - lines.push(`${PREFIX}`); - lines.push(`${PREFIX} ${context.error}`); + lines.push(''); + lines.push(`Error: ${context.error}`); } if (Array.isArray(context.details) && context.details.length > 0) { - lines.push(`${PREFIX}`); + lines.push(''); for (const line of context.details) { - lines.push(`${PREFIX} ${line}`); + lines.push(` ${line}`); } } if (context.stats !== undefined) { - lines.push(`${PREFIX} Stats: ${JSON.stringify(context.stats)}`); + lines.push(''); + lines.push(`Stats: ${JSON.stringify(context.stats, null, 2)}`); } - lines.push(`${PREFIX} ${SEP_EQ}`); - lines.push(`${PREFIX} (email not sent — Entra app pending)`); + return lines.join('\n'); +} +/** + * Send an alert email for pipeline errors. + * Never throws — callers do not guard this. + * + * @param {string} subject - Short description; prefixed with [TestDataDB] ALERT: + * @param {object} [context={}] + * @param {string} [context.stage] - Pipeline stage where the alert fired + * @param {string} [context.error] - Error message + * @param {string[]} [context.details] - Additional detail lines + * @param {object} [context.stats] - Stats object + */ +function alert(subject, context) { + context = context || {}; + + // Always log to stderr so the service log captures it regardless of SMTP try { - for (const line of lines) { - process.stderr.write(line + '\n'); - } + const preview = [ + `${PREFIX} ${SEP_EQ}`, + `${PREFIX} ALERT: [TestDataDB] ${subject}`, + `${PREFIX} Host: ${HOST} | Time: ${new Date().toISOString()}`, + ]; + if (context.stage) preview.push(`${PREFIX} Stage: ${context.stage}`); + if (context.error) preview.push(`${PREFIX} Error: ${context.error}`); + if (context.details) for (const d of context.details) preview.push(`${PREFIX} ${d}`); + if (context.stats) preview.push(`${PREFIX} Stats: ${JSON.stringify(context.stats)}`); + preview.push(`${PREFIX} ${SEP_EQ}`); + for (const line of preview) process.stderr.write(line + '\n'); } catch (_) { - // stderr write failure — nothing we can do + // stderr write failure — nothing to do + } + + const text = buildAlertText(subject, context); + // fire-and-forget; caller never awaits notify.alert + sendMail(`[TestDataDB] ALERT: ${subject}`, text).catch(() => {}); +} + +/** + * Send a daily pipeline summary email. + * Called from run-pipeline.ps1 via a separate Node invocation. + * + * @param {object} stats + * @param {number} stats.received + * @param {number} stats.created + * @param {number} stats.updated + * @param {number} stats.unchanged + * @param {number} stats.errors + * @returns {Promise} + */ +async function summary(stats) { + const date = new Date().toISOString().slice(0, 10); + const status = (stats.errors > 0) ? 'FAIL' : 'OK'; + const subject = `[TestDataDB] Daily pipeline ${status} — ${date}`; + + const lines = [ + `Host: ${HOST}`, + `Time: ${new Date().toISOString()}`, + `Date: ${date}`, + `Status: ${status}`, + SEP_DAS, + `Received: ${stats.received}`, + `Created: ${stats.created}`, + `Updated: ${stats.updated}`, + `Unchanged: ${stats.unchanged}`, + `Errors: ${stats.errors}`, + ]; + + process.stderr.write(`${PREFIX} Sending daily summary (${status}) for ${date}\n`); + await sendMail(subject, lines.join('\n')); +} + +// Allow run-pipeline.ps1 to invoke `node notify.js summary ` +if (require.main === module) { + const cmd = process.argv[2]; + if (cmd === 'summary') { + let stats; + try { + stats = JSON.parse(process.argv[3] || '{}'); + } catch (e) { + process.stderr.write(`${PREFIX} Could not parse stats JSON: ${e.message}\n`); + process.exit(1); + } + summary(stats).then(() => process.exit(0)).catch((e) => { + process.stderr.write(`${PREFIX} summary failed: ${e.message}\n`); + process.exit(1); + }); + } else { + process.stderr.write(`${PREFIX} Unknown command: ${cmd}\n`); + process.exit(1); } } -module.exports = { alert }; +module.exports = { alert, summary }; diff --git a/projects/dataforth-dos/datasheet-pipeline/run-pipeline.ps1 b/projects/dataforth-dos/datasheet-pipeline/run-pipeline.ps1 new file mode 100644 index 0000000..7648d4b --- /dev/null +++ b/projects/dataforth-dos/datasheet-pipeline/run-pipeline.ps1 @@ -0,0 +1,145 @@ +# Dataforth Test Datasheet Uploader (daily) +$ErrorActionPreference = 'Stop' +$prod = 'C:\ProgramData\dataforth-uploader' +$logDir = Join-Path $prod 'logs' +$uploadLogDir = Join-Path $prod 'upload-logs' +$nodeExe = 'C:\Program Files\nodejs\node.exe' +New-Item -ItemType Directory -Force -Path $logDir | Out-Null +$stamp = Get-Date -Format 'yyyy-MM-dd_HH-mm-ss' +$log = Join-Path $logDir "pipeline-$stamp.log" +$smtpCred = $null + +function Log([string]$m) { + $line = "[$(Get-Date -Format o)] $m" + Write-Host $line + Add-Content -Path $log -Value $line -Encoding utf8 +} + +function SendEmail([string]$Subject, [string]$Body) { + if ($null -eq $smtpCred) { return } + try { + Send-MailMessage ` + -From 'sysadmin@dataforth.com' ` + -To @('mike@azcomputerguru.com') ` + -Subject $Subject ` + -Body $Body ` + -SmtpServer 'smtp.office365.com' ` + -Port 587 ` + -UseSsl ` + -Credential $smtpCred + Log "Email sent: $Subject" + } catch { + Log "WARNING: Email send failed: $_" + } +} + +try { + Log "=== pipeline start (pid=$PID) ===" + + # Load credentials + $creds = Get-Content (Join-Path $prod 'credentials.json') -Raw | ConvertFrom-Json + $env:CF_TOKEN_URL = $creds.CF_TOKEN_URL + $env:CF_API_BASE = $creds.CF_API_BASE + $env:CF_CLIENT_ID = $creds.CF_CLIENT_ID + $env:CF_CLIENT_SECRET = $creds.CF_CLIENT_SECRET + $env:CF_SCOPE = $creds.CF_SCOPE + + if ($creds.SMTP_USER -and $creds.SMTP_PASS) { + $secPass = ConvertTo-SecureString $creds.SMTP_PASS -AsPlainText -Force + $smtpCred = New-Object System.Management.Automation.PSCredential($creds.SMTP_USER, $secPass) + Log "SMTP credentials loaded for $($creds.SMTP_USER)" + } else { + Log "WARNING: SMTP_USER/SMTP_PASS not in credentials.json — email notifications disabled" + } + + # [1] DFWDS process + Log '[1] dfwds-process.js' + $dfwdsJs = Join-Path $prod 'dfwds-process.js' + $out = & $nodeExe $dfwdsJs 2>&1 + $out | ForEach-Object { Log $_ } + + # [2] Enumerate For_Web + Log '[2] enumerate For_Web' + $delta = Join-Path $prod 'delta_for_web_all.txt' + Get-ChildItem 'C:\Shares\webshare\For_Web' -File -Filter *.TXT | + ForEach-Object { + $sn = [System.IO.Path]::GetFileNameWithoutExtension($_.Name) + "$sn|$($_.FullName)|$($_.Length)|$($_.LastWriteTime.ToString('o'))" + } | Set-Content -Path $delta -Encoding ASCII + $count = (Get-Content $delta).Count + Log " enumerated $count files" + + # [3] Upload via Node + Log '[3] upload-delta.js' + $uploadJs = Join-Path $prod 'upload-delta.js' + $out = & $nodeExe $uploadJs --delta $delta --batch 100 2>&1 + $out | ForEach-Object { Log $_ } + + # [4] Daily summary email + Log '[4] sending daily summary email' + if ($smtpCred) { + try { + # Parse totals from most recent upload log; default to zeros if not found + $totals = [PSCustomObject]@{ received=0; created=0; updated=0; unchanged=0; errors=0 } + $latestUploadLog = Get-ChildItem $uploadLogDir -Filter '*.log' -ErrorAction SilentlyContinue | + Sort-Object LastWriteTime -Descending | Select-Object -First 1 + if ($latestUploadLog) { + $totalsLine = Get-Content $latestUploadLog.FullName -ErrorAction SilentlyContinue | + Where-Object { $_ -match '^# totals ' } | Select-Object -Last 1 + if ($totalsLine) { + $totals = ($totalsLine -replace '^# totals\s*','') | ConvertFrom-Json + } + } + $status = if ($totals.errors -eq 0) { 'OK' } else { 'FAIL' } + $dateStr = Get-Date -Format 'yyyy-MM-dd' + $summaryBody = @" +Dataforth test datasheet daily pipeline completed on $env:COMPUTERNAME. +Time: $(Get-Date -Format o) + +Files processed : $($totals.received) + Created (new on website) : $($totals.created) + Updated (refreshed) : $($totals.updated) + Unchanged : $($totals.unchanged) + Errors : $($totals.errors) + +Log: $($latestUploadLog.FullName) +"@ + SendEmail "[TestDataDB] Daily pipeline $status — $dateStr" $summaryBody + } catch { + Log "WARNING: Summary email failed: $_" + } + } else { + Log ' SMTP not configured — skipping summary email' + } + + Log '=== pipeline end (OK) ===' +} catch { + Log "FATAL: $_" + Log "StackTrace: $($_.ScriptStackTrace)" + + $date = Get-Date -Format 'yyyy-MM-dd' + $host_ = $env:COMPUTERNAME + $errBody = @" +Host: $host_ +Time: $(Get-Date -Format o) +Date: $date + +FATAL ERROR +----------- +$_ + +Stack Trace +----------- +$($_.ScriptStackTrace) + +Log file: $log +"@ + SendEmail "[TestDataDB] PIPELINE FAILURE — $date" $errBody + + throw +} finally { + # Retention: keep 60 days of pipeline logs + Get-ChildItem $logDir -Filter 'pipeline-*.log' -ErrorAction SilentlyContinue | + Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-60) } | + Remove-Item -Force -ErrorAction SilentlyContinue +} diff --git a/projects/dataforth-dos/session-logs/2026-05-12-session.md b/projects/dataforth-dos/session-logs/2026-05-12-session.md new file mode 100644 index 0000000..c6e68f8 --- /dev/null +++ b/projects/dataforth-dos/session-logs/2026-05-12-session.md @@ -0,0 +1,148 @@ +# Dataforth DOS — 2026-05-12 Session Log + +## User +- **User:** Mike Swanson (mike) +- **Machine:** DESKTOP-0O8A1RL +- **Role:** admin +- **Session span:** 2026-05-12 (continuation from prior context) + +--- + +## Session Summary + +This session audited the full Dataforth test datasheet pipeline to verify it was working and to complete the long-pending email notification feature. The session began by reading CONTEXT.md and the 2026-04-15 session log to establish context, then SSHed to AD2 (192.168.0.6) to inspect the live production state directly. + +The audit confirmed the pipeline is healthy: the `testdatadb` service is running, PostgreSQL contains 469,009 unique records with 458,501 live on the Dataforth website, and the daily scheduled task (`DataforthTestDatasheetUploader`) has been running every morning at 02:30 AM without errors. The task ran this morning, processing 7,517 For_Web files (16 created, 9 updated, 7,492 unchanged, 0 errors). The pipeline has two parallel paths: the real-time DB path (import.js → upload-to-api.js → Hoffman API) and the legacy daily path (dfwds-process.js → For_Web → upload-delta.js → Hoffman API). Both are operational. + +Investigation found that `notify.js` was a stub in both the repo and on AD2 — it only logged to stderr and never sent email. The `run-pipeline.ps1` scheduled task script had no email notification at all. Email was identified as the remaining unimplemented feature. `nodemailer` was installed in testdatadb node_modules (`npm install nodemailer`), SMTP credentials were added to `credentials.json`, `notify.js` was fully implemented, and `run-pipeline.ps1` was updated to send a daily summary email via PowerShell `Send-MailMessage` after each pipeline run. + +A live SMTP test was blocked: `sysadmin@dataforth.com` has SMTP AUTH (basic/legacy auth) disabled in Exchange Online, which is a Microsoft security default. The error was `535 5.7.139 SmtpClientAuthentication is disabled for the Mailbox`. All code is deployed and correct — the only blocker is an Exchange Online configuration change that AJ must make (enable "Authenticated SMTP" for sysadmin in Exchange Admin Center), or alternatively registering an Entra app with Mail.Send for the Graph API approach. Until confirmed, email recipients are set to mike@azcomputerguru.com only; John Lehman will be added once the first email is received. + +Documentation was updated: CONTEXT.md rewritten to reflect the current state (PostgreSQL, dual pipeline paths, email status, undocumented 2026-04-22 changes), PROJECT_STATE.md updated with pending tasks and recent change log. The previously existing `TEST-DATASHEET-PROCESS.md` (written 2026-04-15) was confirmed as a comprehensive end-user document covering architecture, data flow, dashboard UI, and troubleshooting — no changes needed. + +--- + +## Key Decisions + +- **Run-pipeline.ps1 uses PowerShell Send-MailMessage directly** (not Node) — PowerShell 5.1 silently strips double quotes from arguments passed to native executables. `$statsJson` containing `{"key":...}` would arrive at `node notify.js summary` as `{key:...}`, failing JSON.parse. PowerShell-native `Send-MailMessage` has no quoting issue and is the right tool for a PS script. +- **notify.js keeps nodemailer** for service-side alerts — called from Node.js (import.js, upload-to-api.js) where argument passing is not an issue. The two paths use different mechanisms. +- **SMTP creds added to existing credentials.json** (`C:\ProgramData\dataforth-uploader\credentials.json`) rather than a separate file. That file is already ACL'd to SYSTEM + Administrators + svc_testdatadb, covers both the scheduled task and the testdatadb service, and is the single credential source of truth for this deployment. +- **TO list = mike only until confirmed** — avoids sending broken or test emails to John Lehman at Dataforth. Will add jlehman@dataforth.com once the first real email is received. +- **`npm install nodemailer --save`** removed 35 packages that were installed outside of package.json. This was safe — service continued to start and respond HTTP 200. The removed packages (including better-sqlite3) were leftover from the SQLite era; the service now uses PostgreSQL exclusively. +- **Undocumented 2026-04-22 changes documented** — import.js, notify.js, upload-to-api.js all had modification timestamps of 2026-04-22 with multiple backup copies. No session log exists for that date. Changes documented in PROJECT_STATE.md as "undocumented session." + +--- + +## Problems Encountered + +- **psql not in PATH on AD2** — `psql` is not in the system PATH for SSH sessions. Worked around by querying DB stats via the Node.js pg module instead: `node -e "var {Pool}=require('pg');..."`. Root cause: PostgreSQL bin dir not in system PATH, only available to the service account. Did not resolve — not needed since Node is available. + +- **npm removed 35 packages** — `npm install nodemailer --save` detected package.json and cleaned up node_modules to match, removing packages that were manually installed but not in package.json (including better-sqlite3, which was from the SQLite era). Service remained healthy. Resolution: confirmed HTTP 200 after restart, verified all current dependencies (pg, express, pdfkit, nodemailer) present. + +- **PowerShell 5.1 double-quote stripping** — When calling `& $node $script summary $statsJson` in PowerShell, the JSON string `{"received":7517,...}` arrived at Node as `{received:7517,...}` (property names unquoted), causing JSON.parse to fail. Root cause: PowerShell 5.1 strips double quotes when passing arguments to native executables. Resolution: moved summary email to native PowerShell `Send-MailMessage`, eliminating the Node call from run-pipeline.ps1 entirely for the email path. + +- **SMTP AUTH disabled on sysadmin@dataforth.com** — `Send-MailMessage` returned `535 5.7.139 Authentication unsuccessful, SmtpClientAuthentication is disabled for the Mailbox`. This is an Exchange Online security setting. Resolution: UNRESOLVED — needs AJ to enable "Authenticated SMTP" for sysadmin in Exchange Admin Center (`https://admin.exchange.microsoft.com → Mailboxes → sysadmin → Manage mail flow settings → Authenticated SMTP`). Alternative: Entra app with Mail.Send permission + Graph API. + +--- + +## Configuration Changes + +**Repo (D:\claudetools):** +- `projects/dataforth-dos/database/notify.js` — replaced stub with nodemailer implementation. Functions: `alert(subject, ctx)` (fire-and-forget), `summary(stats)` (returns Promise). TO: mike@azcomputerguru.com only (add jlehman once confirmed). SMTP creds loaded from credentials.json at call time. +- `projects/dataforth-dos/datasheet-pipeline/run-pipeline.ps1` — new file in repo. Full PS script for DataforthTestDatasheetUploader task. Added: SMTP cred loading, `SendEmail` function, step [4] summary email after upload, failure alert in catch block. +- `projects/dataforth-dos/CONTEXT.md` — major rewrite. Updated to reflect PostgreSQL DB (was SQLite), dual pipeline paths, email status, as-of-2026-05-12 state. +- `projects/dataforth-dos/PROJECT_STATE.md` — updated current state, pending tasks (email BLOCKER prominent), recent changes table. + +**On AD2 (192.168.0.6):** +- `C:\ProgramData\dataforth-uploader\credentials.json` — added `SMTP_USER: "sysadmin@dataforth.com"` and `SMTP_PASS: "Paper123!@#"` +- `C:\Shares\testdatadb\database\notify.js` — deployed new nodemailer implementation (was stub) +- `C:\ProgramData\dataforth-uploader\run-pipeline.ps1` — deployed updated version with email (was no-email version) +- `C:\Shares\testdatadb\node_modules\nodemailer` — installed (npm install nodemailer) +- `C:\Shares\testdatadb\package.json` — `nodemailer` added to dependencies + +--- + +## Credentials & Secrets + +- **AD2 sysadmin:** `Paper123!@#` — vault: `clients/dataforth/ad2.sops.yaml` → `credentials.password` +- **M365 SMTP:** user=`sysadmin@dataforth.com`, pass=`Paper123!@#` — vault: `clients/dataforth/m365.sops.yaml` → `credentials.password`. SMTP AUTH currently disabled in Exchange Online. +- **Hoffman API OAuth2 creds** (in `credentials.json` on AD2, NOT in vault): + - `CF_TOKEN_URL`: `https://login.dataforth.com/connect/token` + - `CF_API_BASE`: `https://www.dataforth.com` + - `CF_CLIENT_ID`: `dataforth.onprem.sync` + - `CF_CLIENT_SECRET`: `Trxvwee2234-Awer8723-2` + - `CF_SCOPE`: `dataforth.web` +- **PostgreSQL on AD2:** `PGHOST=localhost PGPORT=5432 PGUSER=testdatadb_app PGDATABASE=testdatadb` — password in service environment config + +--- + +## Infrastructure & Servers + +| Component | Details | +|---|---| +| AD2 | 192.168.0.6, Windows Server 2022, sysadmin / Paper123!@# | +| testdatadb service | Node.js v20.10.0, port 3000, runs as INTRANET\svc_testdatadb | +| PostgreSQL | Local on AD2, v18 (approx), PGUSER=testdatadb_app, PGDATABASE=testdatadb | +| Scheduled task | `DataforthTestDatasheetUploader`, daily 02:30 AM, runs as SYSTEM | +| Task files | `C:\ProgramData\dataforth-uploader\` (run-pipeline.ps1, dfwds-process.js, upload-delta.js, credentials.json) | +| Service files | `C:\Shares\testdatadb\` (Node.js app) | +| For_Web | `C:\Shares\webshare\For_Web\` — 7,517 .TXT files | +| Test_Datasheets | `C:\Shares\webshare\Test_Datasheets\` — staging dir for new datasheets from stations | +| Service logs | `C:\Shares\testdatadb\logs\` (out.log, err.log, wrapper.log) | +| Pipeline logs | `C:\ProgramData\dataforth-uploader\logs\pipeline-*.log` (60-day retention) | +| Upload logs | `C:\ProgramData\dataforth-uploader\upload-logs\upload-*.log` | +| Hoffman API | `https://www.dataforth.com/api/v1/TestReportDataFiles/bulk` | +| SMTP (blocked) | smtp.office365.com:587, STARTTLS, sysadmin@dataforth.com | +| M365 Tenant ID | `7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584` | + +--- + +## Commands & Outputs + +``` +# Daily pipeline run (2026-05-12 02:30 AM) — from pipeline log +[2026-05-12T02:30:02] === pipeline start (pid=10064) === +[2026-05-12T02:30:19] [1] dfwds-process.js — valid=16 renamed=0 bad=0 errors=0 +[2026-05-12T02:30:35] [2] enumerate For_Web — enumerated 7517 files +[2026-05-12T02:36:53] [3] upload-delta.js — received=7517 created=16 updated=9 unchanged=7492 errors=0 elapsed=372.5s +[2026-05-12T02:36:53] === pipeline end (OK) === + +# SMTP test failure +535 5.7.139 Authentication unsuccessful, SmtpClientAuthentication is disabled for the Mailbox. +Visit https://aka.ms/smtp_auth_disabled for more information. [PH8PR20CA0019.namprd20.prod.outlook.com] + +# npm install output +added 1 package, removed 35 packages, and audited 124 packages in 14s +# Service survived — HTTP 200 confirmed after restart + +# Key packages confirmed present post-npm +pg: True, express: True, nodemailer: True, pdfkit: True, better-sqlite3: False (was SQLite era, safe) +``` + +--- + +## Pending / Incomplete Tasks + +1. **[BLOCKER] Enable SMTP AUTH for sysadmin@dataforth.com** — AJ must do this in Exchange Admin Center: https://admin.exchange.microsoft.com → Mailboxes → sysadmin → Manage mail flow settings → Authenticated SMTP → Enable. Takes 30 seconds. Alternative: Entra app with Mail.Send (Graph API, more setup but preferred for long term). + +2. **After SMTP confirmed:** Add `jlehman@dataforth.com` to `TO` in `notify.js` (`projects/dataforth-dos/database/notify.js` line 10) and in `run-pipeline.ps1` `-To` array. Deploy both to AD2. Send final confirmation email to John. + +3. **Clean diagnostic scripts on AD2:** `C:\Shares\testdatadb\database\_*.js` files from the 2026-04-15 session (about 20 files). Safe to delete. + +4. **Clean vault entry:** `ad2.sops.yaml` still has stale backslash in password field. Not urgent (strip with `sed 's/\\//g'` at read time as documented). + +5. **Investigate 2026-04-22 undocumented session:** Someone (likely Mike) made changes to import.js, notify.js, upload-to-api.js on that date with no session log. Changes appear stable but details are unknown. + +--- + +## Reference Information + +- **Exchange Admin Center (SMTP AUTH):** https://admin.exchange.microsoft.com → Mailboxes → sysadmin → Manage mail flow settings → Authenticated SMTP +- **SMTP AUTH help link (from error):** https://aka.ms/smtp_auth_disabled +- **Hoffman API bulk upload:** `POST https://www.dataforth.com/api/v1/TestReportDataFiles/bulk` +- **Hoffman OAuth2 token:** `POST https://login.dataforth.com/connect/token` +- **testdatadb dashboard:** http://192.168.0.6:3000/ +- **Scheduled task name:** `DataforthTestDatasheetUploader` +- **AJ contact email:** dataforthgit@ (forwards to AJ) +- **Session logs:** `projects/dataforth-dos/session-logs/` +- **Process docs:** `projects/dataforth-dos/TEST-DATASHEET-PROCESS.md` diff --git a/projects/msp-tools/guru-rmm b/projects/msp-tools/guru-rmm index d0acde5..90d9f60 160000 --- a/projects/msp-tools/guru-rmm +++ b/projects/msp-tools/guru-rmm @@ -1 +1 @@ -Subproject commit d0acde511a1f9d8652bcfb5d26a64a814d8670f0 +Subproject commit 90d9f60710a5d14fac564638794a0d080e46b4bf