sync: auto-sync from DESKTOP-0O8A1RL at 2026-05-12 06:47:00

Author: Mike Swanson
Machine: DESKTOP-0O8A1RL
Timestamp: 2026-05-12 06:47:00

projects/dataforth-dos/CONTEXT.md

@@ -1,7 +1,7 @@
 # Dataforth DOS Project - Context
 
-**Last Updated:** 2026-04-14
-**Status:** Active - Datasheet Pipeline Extended for SCMVAS/SCMHVAS
+**Last Updated:** 2026-05-12
+**Status:** Active - Email notifications implemented (pending M365 SMTP AUTH config)
 
 ## Quick Start - Infrastructure Overview
 
@@ -25,26 +25,35 @@ bash D:/vault/scripts/vault.sh get-field clients/dataforth/ad1.sops.yaml credent
 
 ## Current State (READ THIS FIRST)
 
-### Recent Work (2026-04-11/12)
-**Extended Test Datasheet Pipeline for SCMVAS-Mxxx and SCMHVAS-Mxxxx families**
-- Added VASLOG parser support (multiline CSV .DAT format)
-- Created accuracy-only datasheet template (simple format, no hvin.dat lookup)
-- Implemented pass-through for Engineering-Tested .txt files
-- **Backfilled 27,503 historical records** (438 required regex patch for QB STR$() format quirk)
-- **434 Engineering .txt files** imported and published
-- Deployed to AD2, service restarted, web publishing verified
+### As of 2026-05-12
 
-**Status:** ✅ Complete, production-deployed
+**Production pipeline is healthy and fully operational.** Daily task runs at 02:30 AM, no errors. testdatadb service running on AD2.
 
-**Critical Files Changed:** 5 modified, 1 new parser
-- server/parsers/vaslog.js (new)
-- server/templates/datasheet-exact.js (SCMVAS/SCMHVAS branch added)
-- server/database/import.js (recursive flag fix, VASLOG_ENG support)
-- server/parsers/spec-reader.js (stub for SCMVAS/SCMHVAS)
-- deploy/deploy-to-ad2.py (vault-based credentials)
+**Email notification status:** Code is deployed and wired. Blocked by M365 config — `sysadmin@dataforth.com` has SMTP AUTH (basic auth) disabled in Exchange Online. **Action required from AJ:** Enable "Authenticated SMTP" for sysadmin in Exchange Admin Center, OR create an Entra app with Mail.Send permission for Graph API approach. Recipients once confirmed: jlehman@dataforth.com + mike@azcomputerguru.com. Currently set to mike only for testing.
+
+**DB stats (as of 2026-04-15 release):**
+- 469,009 unique SNs in PostgreSQL
+- 458,501 live on Dataforth website
+- 10,508 internal only (7,905 missing specs, 2,426 Hoffman errors, 177 FAIL)
+- 7,517 For_Web files (legacy path, still used by scheduled task)
+
+### Pipeline Architecture (two parallel paths)
+
+1. **Real-time (testdatadb service):** .dat files → import.js → PostgreSQL → upload-to-api.js → Hoffman API. notify.alert() fires on errors.
+2. **Daily scheduled task (02:30 AM):** dfwds-process.js moves Test_Datasheets → For_Web → upload-delta.js → Hoffman API. Sends summary email on completion.
+
+### Recent Work Summary
+
+**2026-04-15:** Major release — DB dedup (2.89M→469K rows), FAIL→PASS retest rule, For_Web filesystem dependency eliminated (upload-to-api.js now renders in memory), bulk push of 170,984 records to Hoffman, dashboard UI upgrades (pink tint, push buttons, bulk push, website status filter).
+
+**2026-04-22 (undocumented):** Modifications to import.js, notify.js, upload-to-api.js on AD2. No session log exists for this date.
+
+**2026-05-12:** Email notification implementation — nodemailer deployed, credentials.json updated with SMTP creds, run-pipeline.ps1 updated for daily summary email, TEST-DATASHEET-PROCESS.md created as full documentation. Blocked on M365 SMTP AUTH (see above).
 
 **Session Logs:**
-- **2026-04-12-session.md** - Implementation, deploy, backfill, patch (DEFINITIVE)
+- **2026-05-12-session.md** - Email implementation, pipeline audit (CURRENT)
+- **2026-04-15-session.md** - Major release (DB dedup, push, UI)
+- **2026-04-12-session.md** - SCMVAS/SCMHVAS implementation (DEFINITIVE)
 - **2026-04-11-discovery-session.md** - Discovery phase
 
 ### testdatadb Service (on AD2)