From 63109d9033cea23865599c9be4f27a6be712cb3a Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Sun, 24 May 2026 17:56:25 -0700 Subject: [PATCH] wiki: seed Dataforth client + dataforth-dos project articles MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit wiki/clients/dataforth.md — 278 lines: prepaid block contract, all servers/IPs, full contact table, M365/CA policy details, GuruRMM enrollment, patterns (RDS/SAGE-SQL quirks, AD anomalies, C2 iptables not persistent, Win7 EOL), security incident history table. wiki/projects/dataforth-dos.md — 474 lines: DOS update system + TestDataDB pipeline, PostgreSQL schema, FAIL→PASS retest rule, H-prefix decode table, security incident (DF-JOEL2/MFA/IC3), D2TESTNAS role, Neptune SBR email routing, Hoffman API, all anti-patterns. wiki/index.md — Dataforth added to Clients + Projects tables and Cross-Reference; d2testnas added to compilation queue. Co-Authored-By: Claude Sonnet 4.6 --- wiki/clients/dataforth.md | 278 +++++++++++++++++++ wiki/index.md | 6 +- wiki/projects/dataforth-dos.md | 474 +++++++++++++++++++++++++++++++++ 3 files changed, 757 insertions(+), 1 deletion(-) create mode 100644 wiki/clients/dataforth.md create mode 100644 wiki/projects/dataforth-dos.md diff --git a/wiki/clients/dataforth.md b/wiki/clients/dataforth.md new file mode 100644 index 0000000..40a8e27 --- /dev/null +++ b/wiki/clients/dataforth.md @@ -0,0 +1,278 @@ +--- +type: client +name: dataforth +display_name: Dataforth Corporation +last_compiled: 2026-05-24 +compiled_by: DESKTOP-0O8A1RL/claude-main +sources: + - clients/dataforth/docs/overview.md + - clients/dataforth/docs/active-directory.md + - clients/dataforth/docs/workstations.md + - clients/dataforth/docs/manufacturing.md + - clients/dataforth/docs/billing-log.md + - clients/dataforth/docs/SYNC_SCRIPT_UPDATE_SUMMARY.md + - clients/dataforth/docs/incident-2026-03-27-abuse-report-virtuo.md + - clients/dataforth/docs/incident-2026-03-27-abuse-report-connectwise.md + - clients/dataforth/session-logs/2026-03-23-galactic-advisors-report.md + - clients/dataforth/session-logs/2026-03-27-security-incident-mfa-datasheets.md + - clients/dataforth/session-logs/SESSION-SUMMARY.md + - clients/dataforth/session-logs/MEMORY.md + - clients/dataforth/session-logs/2026-04-12-session.md + - clients/dataforth/session-logs/2026-04-13-session.md + - clients/dataforth/session-logs/2026-04-14-session.md + - clients/dataforth/session-logs/2026-04-23-session.md + - clients/dataforth/session-logs/2026-05-03-session.md + - clients/dataforth/session-logs/2026-05-04-lobby-phone-vlan-fix.md + - clients/dataforth/session-logs/2026-05-06-session.md + - clients/dataforth/session-logs/2026-05-12-session.md + - clients/dataforth/session-logs/project_ad2_context.md + - clients/dataforth/session-logs/project_pipeline_rebuilt.md + - clients/dataforth/session-logs/project_test_datasheet_pipeline.md + - clients/dataforth/session-logs/project_new_product_lines.md + - projects/dataforth-dos/CONTEXT.md + - .claude/memory/project_dataforth_incident_2026-03-27.md + - .claude/memory/project_datasheet_pipeline.md + - .claude/memory/project_neptune_sbr_email_routing.md + - .claude/memory/reference_dataforth_contact.md + - .claude/memory/reference_neptune_access_d2testnas.md + - .claude/memory/feedback_d2testnas_ssh.md + - .claude/memory/infra_office_network.md +backlinks: + - projects/dataforth-dos + - systems/jupiter +--- + +# Dataforth Corporation + +Signal conditioning / data acquisition manufacturer in Tucson, AZ. Long-standing ACG client. Active managed relationship — monthly prepaid block. Notable for 64 MS-DOS 6.22 test stations, a major security incident in March 2026, and an ongoing test datasheet pipeline modernization project. + +--- + +## Profile + +- **Contract type:** Prepaid hour block (monthly replenishment invoice $2,098.87) +- **Key contacts:** + +| Name | Username | Role | Email | +|---|---|---|---| +| Dan Center | dcenter | Operations (primary IT contact) | dcenter@dataforth.com | +| John Lehman | jlehman | Engineering, QB code, test specs | jlehman@dataforth.com | +| Peter Iliya | pIliya | Applications Engineer | pIliya@dataforth.com | +| Georg Haubner | ghaubner | Engineering; D: drive on HGHAUBNER has pre-ransomware-attack backup | ghaubner@dataforth.com | +| Kevin Wackerly | kwackerly | IT/Admin, handles calibration@ account | kwackerly@dataforth.com | +| Logan Tobey | ltobey | Support/Sales | ltobey@dataforth.com | +| Ben Wadzinski | bwadzinski | Engineering | — | +| Lee Payne | lpayne | Engineering | — | +| Theresa Dean | tdean | Admin | tdean@dataforth.com | +| Joel Lohr | jlohr | **RETIRED 2026-03-31** — account intentionally kept enabled; inbox rule forwards ntirety.com notifications to mike@azcomputerguru.com | jlohr@dataforth.com | +| Ken Hoffman | khoffman / oemdata | TestDataSheetUploader author, external; also owns Dataforth product API | — | + +- **External distributor:** Ginger (gy@quatronix-cn.com) — Quatronix China; receives datasheets +- **Billing rate:** Prepaid block; all invoices show $0.00 — hours drawn from block +- **Hours remaining:** 46.5 hrs as of 2026-05-03 (after 1 hr billed that session). Always live-check Syncro before billing — `GET /customers/578095`. +- **Syncro customer ID:** 578095 + +--- + +## Infrastructure + +### Servers & Services + +| Host | IP | Role | OS | Notes | +|---|---|---|---|---| +| AD1 | 192.168.0.27 | Primary DC, DNS, FSMO roles, Engineering share | Windows Server 2016 | C:\ at **90%** capacity (C:\Engineering = 787 GB) — critical risk. FSMO roles (assumed all). | +| AD2 | 192.168.0.6 | Secondary DC, TestDataDB service host, NAS mirror, WebShare | Windows Server 2022 | Hosts testdatadb Node.js service on :3000. Wiped by crypto attack 2025 — rebuilt. Windows Firewall disabled (all profiles). | +| FILES-D1 | — | File server | — | Sales docs (W:), archive (Y:) | +| SAGE-SQL | 192.168.0.153 | Sage ERP (S:), RDS Session Host/Connection Broker/Web Access | Windows Server | RDS licensing grace period was expired (reset 2026-05-06). TSGateway disabled (server not externally exposed). New self-signed RDS cert installed. Bitdefender GravityZone managed AV. | +| 3CX | 192.168.0.125 | Phone system | — | Last logon Oct 2025 — possibly inactive | +| DF-HYPERV-B | — | Hyper-V hypervisor | — | — | +| D2TESTNAS | 192.168.0.9 | SMB1 bridge for DOS test stations; Neptune Exchange physically colocated | Linux (CachyOS) | Runs rsync daemon on port 873 (module: `test`, user: `rsync`). SMB1 only — required for DOS 6.22 stations. SSH: `root@192.168.0.9`. Also provides Tailscale route for 172.16.0.0/22 to reach ACG office LAN. | +| ESXi hosts | 192.168.0.122, 192.168.0.124 | VMware ESXi hypervisors | ESXi | — | +| UDM Firewall | 192.168.0.254 | Perimeter firewall/router | UniFi OS | MAC d0:21:f9:6c:11:02. Also responds on 192.168.0.1. SSH key: `~/.ssh/id_ed25519_udm`. C2 IPs blocked via iptables (NOT permanent — need to add to UniFi UI). | +| PBX (3CX/Sangoma) | 192.168.100.2 (also .196) | VoIP PBX — production phones on 192.168.100.0/24 | — | TFTP provisioning for Cisco SPA502G phones. Access via SSH: `sangoma@192.168.100.2`. Vault: `clients/dataforth/pbx.sops.yaml` | + +**Neptune Exchange (ACG infrastructure, physically at Dataforth D2):** +- `neptune.acghosting.com` | internal `172.16.3.11` | external inbound `67.206.163.124` / outbound `67.206.163.122` +- Exchange Server 2016, active ACG-hosted mail server for multiple clients +- Physically colocated at Dataforth's D2 facility — NOT on ACG office LAN despite 172.16.x.x IP +- Access requires routing through D2TESTNAS (192.168.0.9): Dataforth UDM has a 172.16.x.x subnet that overlaps ACG office LAN, making direct routing ambiguous +- SNAT rule on Dataforth UDM at `/data/on_boot.d/10-neptune-snat.sh` should force Neptune outbound to use `.124` (not always active — verify) +- Vault: `clients/dataforth/neptune-exchange.sops.yaml` +- [WARNING] TODO: Resubnet Dataforth UDM to a non-overlapping range to permanently fix Neptune routing + +### Workstations (summary) + +| Category | Count | OS | Notable | +|---|---|---|---| +| Engineering | ~12 | Win 10/11 Pro | HGHAUBNER (192.168.0.148) has pre-attack D: backup. D1-PWRM for PWRM10 test. | +| Manufacturing/Assembly | ~14 | Win 10/11 Pro | AS24, AS26 + various assembly/hi-pot stations | +| Office/Admin | ~12 | Win 10/11 Pro | DF-GAGETRAK (192.168.0.102) — GAGEtrak calibration host. DF-JOEL2 (192.168.0.174) — compromised 2026-03-27, remediated. | +| End-of-Life (Win 7) | 3 | Windows 7 Pro | LABELPC (192.168.0.100), LABELPC2 (192.168.0.98), D2-RCVG-003 (192.168.0.47) — EOL, on network | +| DOS Test Stations | 64 | MS-DOS 6.22 | TS-1 through TS-30 + variants. Not domain-joined. SMB1 via D2TESTNAS. | + +### Email & Identity + +- **M365 tenant:** dataforth.com | Tenant ID: `7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584` +- **Entra ID Sync:** Yes — Azure AD Connect from OU=SyncedUsers only +- **M365 licenses:** 50x Business Premium (39 used), 19x Exchange Online Plan 1 (5 used), 5x SPB (4 used) +- **SMTP settings:** smtp.office365.com, port 587, STARTTLS — use `sysadmin@dataforth.com` +- **SMTP AUTH status:** Tenant-level not disabled; per-mailbox varies. `calibration@dataforth.com` had SmtpClientAuthentication=true re-enabled 2026-04-23. `sysadmin@dataforth.com` SMTP AUTH is blocked by Exchange Online default — testdatadb uses Graph API for email (Mail.Send permission granted to Claude-Code-M365 app 2026-05-12). +- **DKIM:** Both selector1 and selector2 published. Rotated 2026-05-12; cutover to selector2 on 2026-05-16. + - `selector1._domainkey.dataforth.com` → selector1-dataforth-com._domainkey.dataforthcom.onmicrosoft.com + - `selector2._domainkey.dataforth.com` → selector2-dataforth-com._domainkey.dataforthcom.onmicrosoft.com +- **DNS Host:** ntirety.com — Dataforth's public DNS zone managed through ntirety's portal (not a standard registrar). DNS change requests go to ntirety, not a domain control panel. Joel Lohr's account retained to receive ntirety.com infrastructure notifications (inbox rule → mike@azcomputerguru.com). +- **INKY PhishFence:** Active transport rule `B859327F-3FBD-4BE7-A47A-97D02F1558A7` fires first and calls StopProcessingRules=true — blocks all subsequent custom transport rules. Use inbox rules for per-user mail routing. +- **MFA:** 3 Conditional Access policies created 2026-03-27 (initially report-only; enforced 2026-04-04): + - "ACG - Require MFA for All Users" — skip from office IP 67.206.163.122 + - "ACG - Block Foreign Sign-Ins" — US-only; MFA-Travel-Bypass group for exceptions + - "ACG - Block Legacy Authentication" +- **Named locations:** Dataforth Office - Tucson (67.206.163.122/32, trusted), Allowed Countries - US Only +- **MFA-Excluded-BreakGlass group:** Brian Faires, Dataforth Calibration, Dataforth Notifications, Endcap, Tablet 01 +- **MFA enrollment (as of 2026-03-27):** 19/38 ready, 19 needed setup — deadline April 4, 2026 + +### Network + +- **Domain:** intranet.dataforth.com | Forest/Domain Level: Windows Server 2016 +- **ISP:** fdtnet.net | Public IP: 67.206.163.122 (outbound), 67.206.163.124 (Neptune inbound) +- **Firewall/Router:** UniFi Dream Machine at 192.168.0.254 (also 192.168.0.1) +- **Network:** Flat (no VLANs on main LAN — 192.168.0.0/24). Voice/PBX VLAN: 192.168.100.0/24 — production phones live here. UDM default voice VLAN (192.168.1.0/24) not wired to PBX. +- **VPN:** FortiClient required for remote access to 192.168.0.x. VPN can drop mid-session — save work frequently. +- **Drive mappings (GPO):** B: (\\ad1\itsvc), Q: (\\ad2\c-drive), S: (\\SAGE-SQL\sage), T: (\\ad2\e-drive), W: (\\files-d1\sales), X: (\\ad2\webshare), Y: (\\files-d1\archive). DOS test stations: T: (\\D2TESTNAS\test), X: (\\D2TESTNAS\datasheets) + +### GuruRMM Enrollment + +- **Site name:** Dataforth D1 | Site ID: `3a2f6866-26cd-452c-9806-a8df21475c3c` +- **Site API key:** vault `clients/dataforth/...` [check vault for current entry] +- **DF-GAGETRAK enrolled:** Agent ID `7626d82c-0736-47a6-8bc6-68e39859caed`, device ID `win-901ce38b-fb6e-44b8-a577-7c0bdf269a9a` — enrolled 2026-04-23 +- **[WARNING] GuruRMM enrollment workaround:** WebSocket auth in `ws/mod.rs` does not validate `enrolled_agents.agent_key_hash`. New agent installs must overwrite registry AgentKey with the site API key (not the enrollment AgentKey) and restart service. See Gitea issue #8. + +### Key Applications + +| Application | Host | URL/Port | Notes | +|---|---|---|---| +| TestDataDB | AD2 | http://192.168.0.6:3000 | Node.js + Express, PostgreSQL 18, 469K records. Internal LAN only. | +| Sage ERP | SAGE-SQL | \\SAGE-SQL\sage (S:) | RDS-served RemoteApp | +| GageTrak | DF-GAGETRAK (192.168.0.102) | — | Calibration tracking. Sends email via calibration@dataforth.com (SMTP). GuruRMM enrolled. | +| Dataforth Product API | Hoffman's servers | https://www.dataforth.com/api/v1/TestReportDataFiles | OAuth2 client_credentials. Vault: `clients/dataforth/api-oauth.sops.yaml` | +| QuickBASIC 4.5 ATE | 64 DOS stations | T:\ (\\D2TESTNAS\test) | Automated test equipment programs. 1,470+ product model specs. | + +--- + +## Access + +### Domain / Server Access +- **AD2 SSH:** `ssh sysadmin@192.168.0.6` (port 22) — vault: `clients/dataforth/ad2.sops.yaml` → `credentials.password` — NOTE: stale backslash escape in vault entry; strip with `sed 's/\\//g'` +- **AD1 SSH:** `ssh sysadmin@192.168.0.27` — vault: `clients/dataforth/ad1.sops.yaml` +- **D2TESTNAS SSH:** `ssh root@192.168.0.9` — vault: `clients/dataforth/d2testnas.sops.yaml`. Use root, NOT sysadmin (sysadmin SSH fails on D2TESTNAS). SSH key from acg-guru-5070 authorized. +- **UDM SSH:** `ssh root@192.168.0.254` — SSH key `~/.ssh/id_ed25519_udm` (generated 2026-03-27) +- **SAGE-SQL SSH:** `ssh sysadmin@192.168.0.153` — SSH key (`C:\ProgramData\ssh\administrators_authorized_keys` on SAGE-SQL) +- **All server passwords:** `Paper123!@#` (domain admin sysadmin account — stored in individual vault entries per server) +- **WinRM (AD2/AD1):** port 5985 — pywinrm with NTLM, user `INTRANET\sysadmin` + +### M365 / Entra +- **M365 admin:** sysadmin@dataforth.com — vault: `clients/dataforth/m365.sops.yaml` +- **Tenant ID:** `7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584` +- **Claude-Code-M365 Entra App:** App ID `7a8c0b2e-57fb-4d79-9b5a-4b88d21b1f29`, secret expires 2027-12-22 — vault: `clients/dataforth/m365.sops.yaml → credentials.entra-app` +- **MSP Multi-Tenant App (Claude-MSP-Access):** MSP tenant `ce61461e-81a0-4c84-bb4a-7b354a9a356d`, App ID `fabb3421-8b34-484b-bc17-e46de9703418` — vault: msp-tools SOPS file +- **ComputerGuru tiered apps:** All 5 apps consented 2026-04-23. Exchange Operator SP (b43e7342) had Exchange Admin role added manually (gap in onboard-tenant.sh — not auto-assigned for Exch Operator). + +### Dataforth Product API (Hoffman) +- **Vault:** `clients/dataforth/api-oauth.sops.yaml` +- Token URL: `https://login.dataforth.com/connect/token` +- Grant: `client_credentials`, Client ID: `dataforth.onprem.sync`, Scope: `dataforth.web` +- Token TTL: 1 hour +- Swagger: `https://www.dataforth.com/swagger/index.html` + +### ESXi / Hypervisors +- ESXi-122: 192.168.0.122 — vault: `clients/dataforth/esxi-122.sops.yaml` +- ESXi-124: 192.168.0.124 — vault: `clients/dataforth/esxi-124.sops.yaml` + +### PBX +- Vault: `clients/dataforth/pbx.sops.yaml` + +--- + +## Patterns & Known Issues + +### Active Directory +- **No custom security groups** — only default Windows groups. Service accounts in OU=ServiceAccounts. +- **ClaudeTools-ReadOnly AD account** — purpose unclear. Investigate. +- **Ken Hoffman has two accounts** (khoffman + oemdata) — not consolidated. +- **jlohr account retained** — post-retirement (2026-03-31), kept enabled specifically to receive ntirety.com infrastructure notifications. Inbox rule forwards to mike@azcomputerguru.com. Do NOT disable. +- **Entra sync scope:** Only OU=SyncedUsers syncs to Entra. CompanyUsers OU does NOT sync. 38 stale TS-* test station accounts were cleaned from Entra 2026-03-27. + +### RDS / SAGE-SQL +- **RDS licensing:** Grace period reset 2026-05-06 by deleting GracePeriod registry key. Grace period expires again without proper CALs. Purchase RDS CALs (Per User mode, LicensingType=4). +- **TSGateway:** Disabled on SAGE-SQL (server not externally exposed at firewall). Do NOT re-enable without reason. +- **SSL cert:** Self-signed, subject `CN=sage-sql.intranet.dataforth.com`. Non-domain machines must manually import to Trusted Root + Trusted Publishers. +- **GPO cert distribution:** Not completed (AD2 SYSVOL write blocked from non-domain workstation). Pending. +- **Bitdefender GravityZone:** Managed AV on SAGE-SQL. Can block PowerShell execution — may need temporary disable for admin work. + +### Voice / Phones +- **Production phones VLAN:** 192.168.100.0/24. PBX at .196 / .2. All production phones live here. +- **Unifi default voice VLAN (192.168.1.0/24):** NOT used for production — phones landing here cannot reach PBX. Switch port misconfiguration symptom: phone shows wrong date/time (NTP failure) and no dial tone. +- **D1-Server-Room port 1:** Controls lobby drop → must stay on VLAN 100. Reverted to default once before (2026-05-04 incident). + +### Exchange Online / Email +- **INKY PhishFence StopProcessingRules:** Kills all subsequent transport rules. Use inbox rules for per-mailbox forwarding, NOT transport rules. +- **AutoForwarding blocked by default** (tenant outbound spam policy). If per-user forwarding needed, create scoped HostedOutboundSpamFilterPolicy for that sender with AutoForwardingMode=On. +- **Get-MessageTrace deprecated Sept 2025:** Use Get-MessageTraceV2 and Get-MessageTraceDetailV2 in Exchange PowerShell. + +### GuruRMM Agent Deployment +- **WebSocket auth bug (Issue #8):** enrolled_agents.agent_key_hash is never checked by ws/mod.rs. Workaround: after MSI install, overwrite registry `HKLM:\SOFTWARE\GuruRMM\AgentKey` with the site API key (not enrollment AgentKey), then restart service. +- **rmm-api.azcomputerguru.com must be grey-clouded** (DNS-only, not proxied) — Cloudflare proxy blocks WebSocket. Do NOT re-enable orange cloud. Gitea Issue #9. + +### Security +- **C2 IP blocks are iptables only** — do not survive UDM reboot. Must add to permanent UniFi block list via UI. C2 IPs: 80.76.49.18, 45.88.91.99 (AS399486 Virtuo, Montreal). +- **AD1 disk 90% full** — C:\Engineering = 787 GB of 1023 GB. Risk of replication failures. +- **Windows Firewall disabled on AD2** (all profiles) — known risk, not yet remediated. +- **3 Windows 7 machines on network** (LABELPC, LABELPC2, D2-RCVG-003) — EOL, unpatched. +- **AD1/AD2 on Windows Server 2016** — end of mainstream support. Plan upgrade. +- **Entra ID P2 not licensed** — IdentityRiskyUser risk check returns 403 even with scope consented. Would need P2 upgrade to enable Identity Protection. +- **IdentityRiskyUser.Read.All scope:** Consented to Security Investigator app but unusable (no P2 license). + +--- + +## Active Work + +As of 2026-05-12: + +- **Test Datasheet Pipeline:** Production pipeline healthy. 469K records, 458.5K live on website. Daily task runs 02:30 AM. Email notification deployed but pending SMTP AUTH fix — sysadmin SMTP AUTH disabled in Exchange Online. See `projects/dataforth-dos/CONTEXT.md`. +- **GAGEtrak email (ticket #32142):** calibration@ SMTP re-enabled 2026-04-23. GAGEtrak configured (smtp.office365.com:587, calibration@dataforth.com). Kevin Wackerly verifying schedule on DF-GAGETRAK — expected Monday run appears to run Tuesday. +- **DKIM rotation:** Automatic cutover to selector2 on 2026-05-16 — no action needed; verify signing after that date. +- **jlohr forwarding:** ntirety.com inbox rule active as of 2026-05-12; confirmed delivering to mike@azcomputerguru.com. Defunct transport rule pending cleanup. +- **RDS / SAGE-SQL:** RDS grace period reset. GPO cert distribution pending. RDS CALs purchase needed long-term. +- **28 offline machines** (at time of 2026-03-27 incident) — rescanned status unknown. These should be verified when available. +- **MFA enforcement ongoing** — 19 users were still not enrolled as of April 4 enforcement date; current count unverified. + +--- + +## History Highlights + +| Date | Event | +|---|---| +| 2025 | Crypto/ransomware attack — AD2 wiped and rebuilt, many files lost. Test datasheet pipeline broken. | +| 2026-01-19 | DOS Update System built and deployed — NWTOC/CTONW/UPDATE/DEPLOY BAT files, 39 deployments. Sync-FromNAS updated (DEPLOY.BAT). | +| 2026-03-20 | Galactic Advisors security assessment — AD1 C: at 90%, legacy SQL 2008 R2 client noted, 3 computers scanned. | +| 2026-03-23 | Galactic Advisors assessment analyzed by ACG. | +| 2026-03-27 | **Major security incident:** DF-JOEL2 compromised via social engineering/ScreenConnect (attacker "Angel Raya", C2 on Virtuo hosting). M365 sign-in from Turkey. Full remediation. 3 CA policies deployed. MFA notice sent. IC3 filed (1c32ade367084be9acd548f23705736f). | +| 2026-03-27–29 | Test datasheet pipeline rebuilt — 72/73 Quatronix datasheets generated, new Node.js pipeline replaces VB6 DFWDS + VB.NET uploader. | +| 2026-03-31 | Joel Lohr retirement. Brian Faires mailbox converted to shared (5,711 messages preserved). 38 stale Entra TS-* accounts deleted. | +| 2026-04-04 | MFA CA policies enforced (switched from report-only). | +| 2026-04-11–12 | SCMVAS/SCMHVAS pipeline extension — 27,503 records backfilled, 434 Engineering-Tested .txt files imported. | +| 2026-04-12 | TestDataDB PostgreSQL migration verified (2.89M records). Hoffman API discovered (Swagger). | +| 2026-04-13 | API architecture discussion with Hoffman — client_credentials grant confirmed for dataforth.onprem.sync client. | +| 2026-04-14 | DFWDS logic ported to Node.js (dfwds-process.js). 897 staged datasheets drained. 803 new records created on Hoffman API. | +| 2026-04-15 | Major release — DB dedup (2.89M→469K rows), FAIL→PASS retest rule, For_Web filesystem dependency eliminated, 170,984 records bulk-pushed to Hoffman. Dashboard UI upgrades. | +| 2026-04-23 | Full Dataforth tenant onboarded to all 5 ComputerGuru tiered apps. calibration@ SMTP AUTH fixed. DF-GAGETRAK GuruRMM agent enrolled (with auth workaround). Syncro ticket #32142 billed. | +| 2026-05-03 | jantar@dataforth.com darkweb breach check — no indicators of compromise. eM Client OAuth grant and SP revoked/disabled. 1 hr billed. | +| 2026-05-04 | Howard onsite — lobby phone offline (VLAN misconfiguration on D1-Server-Room port 1 → fixed to VLAN 100). | +| 2026-05-06 | SAGE-SQL RDS issues resolved — grace period reset, SSL cert replaced, TSGateway disabled, RemoteApp permission prompts fixed. | +| 2026-05-12 | Pipeline audit + email notifications implemented (Graph API). jlohr forwarding configured (ntirety.com → mike@). DKIM keys rotated. | + +--- + +## Backlinks + +- [[projects/dataforth-dos]] — Active test datasheet pipeline project on AD2 +- [[systems/jupiter]] — Neptune Exchange physically colocated at Dataforth D2 facility; D2TESTNAS provides Tailscale routing diff --git a/wiki/index.md b/wiki/index.md index 49e4315..c72cbaf 100644 --- a/wiki/index.md +++ b/wiki/index.md @@ -19,12 +19,14 @@ Run `/wiki-lint` to check for stale entries and broken backlinks. | Article | Summary | Last Compiled | |---|---|---| | [Cascades of Tucson](clients/cascades-tucson.md) | Prepaid block $175/hr, ~37.5 hrs remaining; senior living; active domain migration + HIPAA compliance project; single DC on aging R610 hardware | 2026-05-24 | +| [Dataforth Corporation](clients/dataforth.md) | Prepaid block ~$2,099/mo; signal conditioning manufacturer; 64 DOS test stations; 2025 crypto attack recovery; 2026-03-27 phishing incident + MFA rollout; active test datasheet pipeline project; Neptune Exchange colocated at D2 | 2026-05-24 | ## Projects | Article | Summary | Last Compiled | |---|---|---| | [GuruRMM](projects/gururmm.md) | RMM platform, Rust/Axum server + React dashboard + cross-platform agent; v0.6.38; 55 enrolled agents; active development | 2026-05-24 | +| [Dataforth DOS — Test Datasheet Pipeline](projects/dataforth-dos.md) | DOS update system + TestDataDB pipeline (Node.js, PostgreSQL, Hoffman API); 469K records, 458.5K live on website; 2025 crypto attack recovery; security incident 2026-03-27; SCMVAS/SCMHVAS extension; email notifications via Graph API | 2026-05-24 | ## Systems @@ -49,6 +51,7 @@ Run `/wiki-lint` to check for stale entries and broken backlinks. |---|---|---| | Cascades of Tucson | CS-SERVER (192.168.2.254), pfSense (192.168.0.1), cascadesDS (192.168.0.120) | GuruRMM (RECEPTIONIST-PC + CS-SERVER enrolled) | | ACG Internal | gururmm-build (172.16.3.30), Jupiter (172.16.3.20), Pluto (172.16.3.36), Uranus (172.16.3.21) | GuruRMM server + ClaudeTools API on gururmm-build; Windows MSI builds on Pluto; Gitea/NPM/Seafile on Jupiter. Saturn DECOMMISSIONED. | +| Dataforth Corporation | AD1 (192.168.0.27), AD2 (192.168.0.6), D2TESTNAS (192.168.0.9), SAGE-SQL (192.168.0.153), UDM (192.168.0.254); Neptune Exchange physically at Dataforth D2 (172.16.3.11 / 67.206.163.124) | Dataforth DOS — Test Datasheet Pipeline; GuruRMM (DF-GAGETRAK enrolled) | --- @@ -56,6 +59,7 @@ Run `/wiki-lint` to check for stale entries and broken backlinks. | Scope | Priority | Notes | |---|---|---| -| `system:neptune` | Low | neptune.acghosting.com, 172.16.3.11 internal / 67.206.163.124 external — Exchange Server 2016; ACG infrastructure physically colocated at Dataforth D2 facility; active mail server for multiple ACG-hosted clients; internal access requires routing through D2TESTNAS because Dataforth UDM runs a subnet that duplicates/overlaps ACG office LAN (172.16.x.x) — TODO: resubnet Dataforth UDM to eliminate overlap | +| `system:neptune` | Low | neptune.acghosting.com, 172.16.3.11 internal / 67.206.163.124 external — Exchange Server 2016; ACG infrastructure physically colocated at Dataforth D2 facility; active mail server for multiple ACG-hosted clients; Neptune context captured in clients/dataforth.md and projects/dataforth-dos.md; still warrants own system article for SBR config, MailProtector, per-client send connectors, and full routing detail | +| `system:d2testnas` | Low | 192.168.0.9 — Linux (CachyOS?), SMB1 bridge for Dataforth DOS stations, rsync daemon port 873, hosts Neptune Exchange physically; key routing node for ACG-Dataforth connectivity; SSH root@192.168.0.9; also provides Tailscale 172.16.0.0/22 route | | `client:birthbiologic` | Medium | GuruRMM enrolled (site BRIGHT-PEAK-5980) | | `client:key-paul` | Low | GuruRMM enrolled (KEY-MEDIA) | diff --git a/wiki/projects/dataforth-dos.md b/wiki/projects/dataforth-dos.md new file mode 100644 index 0000000..33113c9 --- /dev/null +++ b/wiki/projects/dataforth-dos.md @@ -0,0 +1,474 @@ +--- +type: project +name: dataforth-dos +display_name: Dataforth DOS — Test Datasheet Pipeline +last_compiled: 2026-05-24 +compiled_by: DESKTOP-0O8A1RL/claude-main +sources: + - projects/dataforth-dos/CONTEXT.md + - projects/dataforth-dos/PROJECT_STATE.md + - projects/dataforth-dos/PROJECT_INDEX.md + - projects/dataforth-dos/TEST-DATASHEET-PROCESS.md + - projects/dataforth-dos/Sync-FromNAS.ps1 + - projects/dataforth-dos/session-logs/2026-01-20-session.md + - projects/dataforth-dos/session-logs/2026-01-21-session.md + - projects/dataforth-dos/session-logs/2026-03-11-testdatadb-investigation.md + - projects/dataforth-dos/session-logs/2026-03-12-session.md + - projects/dataforth-dos/session-logs/2026-03-13-import-fix.md + - projects/dataforth-dos/session-logs/2026-03-16-session.md + - projects/dataforth-dos/session-logs/2026-04-11-discovery-session.md + - projects/dataforth-dos/session-logs/2026-04-12-session.md + - projects/dataforth-dos/session-logs/2026-04-15-session.md + - projects/dataforth-dos/session-logs/2026-05-12-session.md + - clients/dataforth/session-logs/2026-03-27-security-incident-mfa-datasheets.md + - clients/dataforth/session-logs/SESSION-SUMMARY.md + - clients/dataforth/session-logs/MEMORY.md + - clients/dataforth/session-logs/2026-04-12-session.md + - clients/dataforth/session-logs/2026-04-13-session.md + - clients/dataforth/session-logs/2026-04-14-session.md + - clients/dataforth/docs/manufacturing.md + - .claude/memory/project_datasheet_pipeline.md + - .claude/memory/project_dataforth_incident_2026-03-27.md + - .claude/memory/feedback_d2testnas_ssh.md +backlinks: + - clients/dataforth + - systems/jupiter +--- + +# Dataforth DOS — Test Datasheet Pipeline + +Dataforth Corporation manufactures signal conditioning / data acquisition modules and has 64 MS-DOS 6.22 automated test stations. This project covers the full infrastructure that routes test data from those DOS stations into a modern web-published test datasheet system. The "DOS" name reflects the project's origin (DOS update system, 2026-01-20) but the scope expanded dramatically to include the TestDataDB service, the Hoffman API integration, and a major security incident response. + +**Current state (2026-05-12):** Production pipeline fully operational. 469K records, 458.5K live on www.dataforth.com. Daily scheduled task runs at 02:30 AM. + +--- + +## Summary + +Dataforth's 64 QuickBASIC 4.5 test programs produce binary test logs for every module that passes through manufacturing. These logs must be converted to formatted "test data sheets" — text documents matching a proprietary QuickBASIC template — and published on the public Dataforth website for customer download. + +The original pipeline involved DOS batch files, a VB6 filename decoder (DFWDS.exe), and a VB.NET web uploader — all of which were destroyed in a 2025 ransomware attack that wiped AD2. ACG rebuilt the pipeline in March 2026 using Node.js and a modern REST API, then has continued iterating: PostgreSQL migration, SCMVAS/SCMHVAS product line support, DB dedup, FAIL→PASS retest logic, direct API upload, and email notifications. + +The project also encompasses the original DOS Update System (batch file infrastructure for over-the-network software updates to the 64 test stations, production-ready since 2026-01-20). + +--- + +## Architecture + +### Component Overview + +``` +DOS Test Stations (64) + MS-DOS 6.22, QuickBASIC 4.5 ATE + Writes binary .DAT files per test run + | + | CTONW.BAT — uploads .DAT via SMB1 + v +D2TESTNAS (192.168.0.9) — Samba SMB1 bridge + rsync daemon port 873, module "test" + /data/test → serves as T:\ for DOS stations + | + | Sync-FromNAS.ps1 — every 15 min on AD2 + v +AD2 (192.168.0.6) — C:\Shares\test\ (NAS mirror) + | + | testdatadb service — import.js + v +PostgreSQL 18 (local on AD2) — testdatadb DB + 469,009 unique records (UNIQUE on serial_number) + | + | upload-to-api.js — real-time after import + | run-pipeline.ps1 — daily 02:30 AM fallback + v +Hoffman Product API — https://www.dataforth.com + /api/v1/TestReportDataFiles/bulk + OAuth2 client_credentials + 458,501 records live on website (as of 2026-04-15) + 661,367 total on Hoffman (includes 202,866 pre-testdatadb historical) +``` + +### Components + +| Component | Location | Tech | State | +|---|---|---|---| +| DOS test stations | Physical (64 stations, TS-1 through TS-30 + variants) | MS-DOS 6.22, QuickBASIC 4.5 | Production | +| D2TESTNAS NAS | 192.168.0.9 | Linux (Samba, rsync daemon) | Production | +| Sync-FromNAS.ps1 | AD2 C:\Shares\test\scripts\ | PowerShell, rsync over SSH | Production — every 15 min | +| testdatadb service | AD2 C:\Shares\testdatadb\ | Node.js v20, Express, WinSW | Production — runs as INTRANET\svc_testdatadb | +| PostgreSQL 18 | AD2 localhost:5432 | PostgreSQL | Production | +| Dashboard UI | http://192.168.0.6:3000/ | Node.js/Express/HTML | Production — internal LAN only | +| Daily scheduled task | AD2 (Task Scheduler) | PowerShell + Node.js | Production — 02:30 AM as SYSTEM | +| Hoffman Product API | https://www.dataforth.com | REST/OAuth2 | Third-party (Ken Hoffman) | +| DOS Update System | AD2 C:\Shares\test\ + D2TESTNAS /data/test/ | DOS batch files, PS deployment | Production since 2026-01-20 | + +### Two Parallel Upload Paths + +**Path 1 — Real-time (preferred):** +import.js ingests new .DAT records → PostgreSQL → upload-to-api.js immediately pushes PASS records to Hoffman API. `api_uploaded_at` stamped on success. + +**Path 2 — Daily scheduled fallback (02:30 AM):** +`DataforthTestDatasheetUploader` task runs `C:\ProgramData\dataforth-uploader\run-pipeline.ps1`. Runs dfwds-process.js (moves Test_Datasheets → For_Web), then upload-delta.js (pushes all For_Web files). Acts as safety net for real-time path failures. Sends summary email via Graph API on completion. + +### Key File Paths + +**On AD2 (192.168.0.6):** + +| Path | Purpose | +|---|---| +| `C:\Shares\testdatadb\` | testdatadb Node.js application root | +| `C:\Shares\testdatadb\database\import.js` | Parses .DAT files, inserts to PostgreSQL | +| `C:\Shares\testdatadb\database\upload-to-api.js` | Pushes records to Hoffman API | +| `C:\Shares\testdatadb\database\render-datasheet.js` | In-memory datasheet rendering (no FS dependency) | +| `C:\Shares\testdatadb\database\export-datasheets.js` | Legacy For_Web .TXT writer (retained for compat) | +| `C:\Shares\testdatadb\parsers\` | Binary .DAT parsers per log type (multiline.js, vaslog.js, etc.) | +| `C:\Shares\testdatadb\specdata\` | QuickBASIC binary spec files (5BMAIN.DAT, 7BMAIN.DAT, etc.) | +| `C:\Shares\testdatadb\templates\datasheet-exact.js` | Datasheet formatter replicating QuickBASIC output | +| `C:\Shares\testdatadb\logs\` | Service logs (out.log, err.log, wrapper.log) | +| `C:\Shares\testdatadb\daemon\testdatadb.exe` | WinSW service wrapper | +| `C:\Shares\webshare\For_Web\` | ~7,517 legacy .TXT datasheet files | +| `C:\Shares\webshare\Test_Datasheets\` | Staging dir for new datasheets from stations | +| `C:\Shares\webshare\Bad_Datasheets\` | ~18,801 invalid/quarantined files (historical) | +| `C:\Shares\webshare\Datasheets_Log\` | DFWDS processing run logs | +| `C:\Shares\test\` | NAS mirror of test station data | +| `C:\Shares\test\Ate\HISTLOGS\` | Central consolidated test logs (per log type, per model) | +| `C:\Shares\test\scripts\Sync-FromNAS-rsync.ps1` | NAS sync script | +| `C:\ProgramData\dataforth-uploader\` | Scheduled task scripts, credentials.json, pipeline logs | +| `C:\ProgramData\dataforth-uploader\credentials.json` | Hoffman API OAuth2 creds + Graph API creds; ACL: SYSTEM + Admins + svc_testdatadb | +| `C:\Users\sysadmin\Documents\dataforth-uploader\` | dfwds-process.js and upload-delta.js (used by daily task) | + +**On AD1 (192.168.0.27):** + +| Path | Purpose | +|---|---| +| `\\AD1\Engineering\ENGR\ATE\` | QuickBASIC source, spec files per product family | +| `\\AD1\Engineering\ENGR\ATE\High Voltage Input Module Test\HVDATA\hvin.dat` | SCMVAS/SCMHVAS spec binary (33 records, engineering MODNAMEs — NOT marketing names) | + +**Repo (D:\claudetools):** + +| Path | Purpose | +|---|---| +| `projects/dataforth-dos/` | Project root | +| `projects/dataforth-dos/datasheet-pipeline/` | Pipeline scripts and research | +| `projects/dataforth-dos/datasheet-pipeline/implementation/` | Production-staged code (deployer: deploy-to-ad2.py) | +| `projects/dataforth-dos/datasheet-pipeline/scmvas-hvas-research/` | Discovery artifacts for SCMVAS/SCMHVAS extension | +| `projects/dataforth-dos/deploy/` | Deployment scripts | +| `projects/dataforth-dos/TEST-DATASHEET-PROCESS.md` | End-user process documentation (audience: Dataforth Engineering) | + +--- + +## Data Model & Datasheet Pipeline Detail + +### PostgreSQL Schema (key table) + +```sql +CREATE TABLE test_records ( + id SERIAL PRIMARY KEY, + log_type VARCHAR(20) NOT NULL, -- 5BLOG, 7BLOG, 8BLOG, DSCLOG, SCTLOG, VASLOG, VASLOG_ENG, etc. + model_number VARCHAR(100) NOT NULL, + serial_number VARCHAR(100) NOT NULL, + test_date DATE, + test_station VARCHAR(50), + overall_result VARCHAR(10), -- 'PASS' or 'FAIL' + raw_data TEXT, -- decoded record content + source_file TEXT, + work_order VARCHAR(50), + datasheet_exported_at TIMESTAMPTZ, + forweb_exported_at TIMESTAMPTZ, -- legacy For_Web file write + api_uploaded_at TIMESTAMPTZ, -- when pushed to Hoffman + import_date TIMESTAMPTZ DEFAULT NOW(), + search_vector tsvector, + CONSTRAINT uq_test_records_sn UNIQUE (serial_number) +); +``` + +### FAIL→PASS Retest Rule + +Engineering directive: if a unit fails, is repaired, and passes a later retest, the PASS record replaces the FAIL. But if a unit already has a PASS and later tests FAIL (e.g., post-ship retest), the PASS is kept. Encoded as `ON CONFLICT (serial_number) DO UPDATE ... WHERE test_records.overall_result = 'FAIL' OR (EXCLUDED.overall_result = 'PASS' AND EXCLUDED.test_date > test_records.test_date)`. + +### Log Types and Spec Files + +| Log Type | Product Family | Spec File(s) | Notes | +|---|---|---|---| +| 5BLOG | SCM5B (isolated signal conditioning) | 5BMAIN.DAT, 5B45DATA.DAT, 5B49_2.DAT, DB5B48.DAT | 481 + 56 + 15 + 3 = 555 models | +| 7BLOG | SCM7B | 7BMAIN.DAT | 276 models | +| 8BLOG | 8B | 8BMAIN.DAT | 148 models | +| DSCLOG | DSCA | DSCMAIN4.DAT, DSCOUT.DAT | 391 + 23 = 414 models | +| SCTLOG | DSCT transmitters | SCTMAIN.DAT | 103 models | +| VASLOG | SCMVAS, SCMHVAS (production ATE path) | No spec file — accuracy-only format | Marketing names (SCMHVAS-M0100 etc.); do NOT look up in hvin.dat | +| VASLOG_ENG | SCMHVAS (engineering-tested variant) | None — verbatim file passthrough | 434 files imported; stored as raw_data, copied byte-exact to output | +| PWRLOG | Power supplies | Specs embedded in parser | — | +| SHT | Short-form records | Specs embedded | — | + +**Total spec models: 1,470+** (stored at C:\Shares\testdatadb\specdata\ on AD2; source: \\AD1\Engineering\ENGR\ATE\\\) + +### H-prefix Filename Decode + +DOS QuickBASIC programs encode serial numbers using a letter prefix for the leading two digits: +A=10, B=11, C=12, D=13, E=14, F=15, G=16, H=17, I=18, J=19. +Example: `H8601-6.TXT` → serial `178601-6`. +**Always extract serial numbers from the DAT record data content, never from the 8.3 filename.** + +### Dashboard Features + +URL: `http://192.168.0.6:3000/` (internal LAN only) + +- Filter by Serial Number, Work Order, Model Number, Result, Product Line, Website Status (Any/On Website/Not on Website), Test Station, Date range, full-text search +- **Pink tint** on rows = record not yet on Dataforth website (api_uploaded_at IS NULL) +- Per-row PUSH / RE-PUSH buttons; bulk "PUSH TO WEB" button +- CSV export of current filter set +- SHEET preview — renders the exact datasheet text that was/would be sent to website + +--- + +## Security Incident History (2026-03-27) + +### DF-JOEL2 Compromise + +Joel Lohr's workstation was compromised via phishing email to his personal Yahoo/Comcast account (appeared to be from Arizona Technology Council). Attacker "Angel Raya" installed ScreenConnect from a phishing link, then deployed two C2 backdoor ScreenConnect clients and used a tool to hide them from the uninstall list. + +**Timeline (2026-03-27 MST):** +- 08:25 — Joel clicks phishing link +- 08:28 — ScreenConnect.ClientSetup.msi downloaded to C:\Users\jlohr\Downloads\ +- 08:29 — "Angel Raya" connected via cloud relay `instance-wlb9ga-relay.screenconnect.com` +- 08:29 — Two C2 backdoor clients deployed via PowerShell +- 08:31 — Sordum "Hide From Uninstall List" tool downloaded +- 08:32 — Rogue clients hidden; "Angel Raya" disconnected +- 11:55 — "Administrator" connected via C2 IP 80.76.49.18 +- 18:51 — Successful unauthorized M365 sign-in from Istanbul, Turkey (91.93.232.236) + +**Attacker Infrastructure:** +- C2 Server 1: 80.76.49.18:8040/8041 +- C2 Server 2: 45.88.91.99:8040/8041 +- ASN: AS399486, Virtuo (12651980 CANADA INC.), Montreal QC +- Abuse: abuses@virtuo.host (automated suspension confirmed) +- Cloud relay: instance-wlb9ga-relay.screenconnect.com +- ConnectWise case: 03464184 + +**Rogue ScreenConnect clients (all removed):** +- `0cad93610010625f` — "Angel Raya" initial access (cloud relay) +- `0dfe1abae029411c` — C2 backdoor (80.76.49.18:8041) +- `a897d9a21259d116` — C2 backdoor (45.88.91.99:8041) +- Legitimate ACG client: `1912bf3444b41a08` (instance-kgc7jt) — NOT removed + +**M365 Compromise (jlohr@dataforth.com):** +- Brute-force for 7+ days; successful logins from Istanbul Turkey, Croydon UK, Germany +- Azure AD PowerShell and Azure CLI used by attacker +- No malicious inbox rules, forwarding, or OAuth consents found + +**Remediation completed 2026-03-27:** +- C2 IPs blocked at UDM firewall (iptables INPUT + FORWARD — not permanent; add to UniFi UI) +- 3 rogue ScreenConnect clients uninstalled via WinRM +- HideUL tool deleted from C:\Users\Public\Pictures\Backup\ +- jlohr AD password reset; Entra sessions revoked +- 32 machines scanned clean (28 unreachable/offline — status unknown) +- No lateral movement detected + +**Reports filed:** +- FBI IC3: Submission ID `1c32ade367084be9acd548f23705736f` (filed 2026-03-27 5:11 PM EST) +- Virtuo Hosting: abuses@virtuo.host +- ConnectWise: Case #03464184 +- Artifacts: `clients/dataforth/docs/IC3-Complaint-2026-03-27.pdf`, `clients/dataforth/docs/incident-2026-03-27-abuse-report-virtuo.md`, `clients/dataforth/docs/incident-2026-03-27-abuse-report-connectwise.md` + +### MFA Deployment (same day) + +Three Conditional Access policies deployed (report-only initially, enforced 2026-04-04): +- `dc920ee4-22e6-402b-b5e3-4f3662d26227` — Require MFA (skip from office IP 67.206.163.122) +- `3405f7db-91b6-48da-b3fb-2e0ef1e44d17` — Block Foreign Sign-Ins (US only) +- `82ebbe3b-d151-4cb7-aff7-af893a4915e3` — Block Legacy Auth + +Named locations: +- `0a3e61d7-a544-4a47-961a-a98cd4804613` — Dataforth Office - Tucson (67.206.163.122/32) +- `12706cec-c91b-454e-a24d-c801284b79f7` — Allowed Countries - US Only + +Security groups: +- `75ac10ae-d49e-42b1-aa87-04908a983495` — MFA-Excluded-BreakGlass +- `094b12c5-b39a-4287-943a-f1175ce61a6f` — MFA-Travel-Bypass + +--- + +## D2TESTNAS — Role and Access + +D2TESTNAS (192.168.0.9) is a Linux machine serving as the critical bridge between the DOS 6.22 test stations and the modern Windows/IP infrastructure. DOS 6.22 only supports SMB1 (not SMB2+), so D2TESTNAS runs Samba as an SMB1 share while also running an rsync daemon for efficient syncing to/from AD2. + +**D2TESTNAS also physically houses Neptune Exchange Server** — ACG's Exchange Server 2016 instance that hosts mail for multiple ACG clients. Neptune's internal IP (172.16.3.11) is within the ACG LAN range, but physically the machine is at Dataforth's D2 facility. Access requires routing through D2TESTNAS because Dataforth's UDM uses a 172.16.x.x subnet that overlaps the ACG office LAN, making direct Tailscale routing to Neptune ambiguous. + +**D2TESTNAS access:** +- SSH: `root@192.168.0.9` — vault: `clients/dataforth/d2testnas.sops.yaml` +- [WARNING] Use `root`, NOT `sysadmin` — `sysadmin` SSH fails on D2TESTNAS +- SSH key from acg-guru-5070 (ed25519) is authorized +- rsync daemon: port 873, module `test`, user `rsync` — vault contains rsync credentials +- Tailscale: provides the 172.16.0.0/22 route from Dataforth network to ACG office + +**Quirk:** Dataforth UDM runs a 172.16.x.x subnet that overlaps ACG's office LAN (172.16.0.0/22). This creates routing ambiguity for direct Tailscale connections to 172.16.3.x (Neptune's range). Until Dataforth UDM is resubnetted, always access Neptune via D2TESTNAS. + +--- + +## Neptune SBR Email Routing + +Neptune Exchange Server routes outbound mail through the MailProtector smarthost via the SBR (Sender-Based Routing) transport agent. This is ACG infrastructure physically hosted at Dataforth D2. + +**Outbound chain:** +1. User sends from Neptune mailbox (172.16.3.11) +2. SBR transport agent (Priority 12) fires on `OnResolved` event +3. SBR reads config files at `C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\agents\Custom\` — `InternalDomains.config`, `OverrideSettings.config`, `IgnoreAuthAs.config` +4. SBR rewrites routing to `.sbr` domain (e.g., `rieussetcorp.sbr`) +5. Exchange matches to send connector → smarthosts via MailProtector (`domain-com.outbound.emailservice.io`) +6. MailProtector relays to final destination + +**Common issue:** When Neptune's IP changes or a new client domain is added, MailProtector must have the sending server IP authorized. Without this, MailProtector silently drops the message. + +**IP:** Neptune outbound uses `67.206.163.122` (sometimes `.124` — SNAT rule on Dataforth UDM should force outbound to `.124` but may not always be active). 67.206.163.122 has no PTR record and is blacklisted by some providers. + +**Neptune access:** Requires Tailscale via D2TESTNAS (see above). WinRM to 172.16.3.11, `ACG\administrator` — vault: `clients/dataforth/neptune-exchange.sops.yaml`. + +--- + +## Dataforth Product API (Hoffman) + +Ken Hoffman built and maintains the public-facing API that serves datasheets on www.dataforth.com. + +**Endpoints of interest:** +- `POST /api/v1/TestReportDataFiles` — single upload `{SerialNumber, Content}` +- `POST /api/v1/TestReportDataFiles/bulk` — batch upload `{Items: [{SerialNumber, Content}, ...]}` +- `GET /api/v1/TestReportDataFiles/{serialNumber}` — retrieve one +- `GET /api/v1/TestReportDataFiles/stats` — `{TotalCount, LatestCreatedAtUtc, LatestUpdatedAtUtc}` +- Swagger: `https://www.dataforth.com/swagger/index.html` + +**Authentication:** OAuth2 client_credentials. Token cached 1 hour. Refresh on 401. +- Token URL: `https://login.dataforth.com/connect/token` +- Client ID: `dataforth.onprem.sync` +- Client Secret: vault `clients/dataforth/api-oauth.sops.yaml` +- Scope: `dataforth.web` + +**Idempotency:** Server deduplicates by content hash. Same SN + same content → `Unchanged`. Same SN + different content → `Updated`. New SN → `Created`. Safe to re-push everything. + +**Throughput observed:** ~142 records/s sustained for bulk uploads (batches of 100). + +**Hoffman API state (2026-04-15):** 661,367 total records on website. 202,866 are pre-testdatadb historical (uploaded by old DFWDS toolchain — not reproducible from current DB, don't try to reconcile). + +--- + +## Known Issues and Anti-Patterns + +### Anti-Patterns (from CONTEXT.md) + +- **DO NOT hardcode Paper123!@#** — always fetch from vault: `bash D:/vault/scripts/vault.sh get-field clients/dataforth/ad2.sops.yaml credentials.password | sed 's/\\//g'` +- **DO NOT use X: drive in SSH sessions** — X: is only mapped under the service account. Use UNC `\\ad2\webshare\For_Web` instead. +- **DO NOT assume hvin.dat model lookup works for VASLOG** — marketing names (SCMHVAS-M0100) do NOT match engineering MODNAMEs (SCM5B41-1181) in hvin.dat. SCMVAS/SCMHVAS use simple accuracy-only template without hvin.dat. +- **DO NOT pass 50+ file paths on PowerShell command line** — hits "Command line too long". Use inline Node.js with `fs.readdirSync` instead. +- **DO NOT commit testdata.db or large samples** — 4.1GB database is in .gitignore. +- **DO NOT use SMB1 on AD2** — disabled for security. Use SSH/SFTP (port 22) or SMB2+ shares. +- **DO NOT expect immediate stdout from paramiko exec_command** — buffers until completion. Use progress markers or drain loop. +- **Vault entry ad2.sops.yaml has stale backslash escape** — password stored as `Paper123\!@#`, actual password is `Paper123!@#`. Strip with `sed 's/\\//g'` at read time until vault is cleaned. +- **DO NOT build GuruRMM agents manually** — all builds via Gitea webhook pipeline (push to main). + +### Known Issues + +- **Email notifications blocked on SMTP AUTH:** sysadmin@dataforth.com has SMTP AUTH disabled by Exchange Online. Email now uses Graph API (Mail.Send granted to Claude-Code-M365 app 2026-05-12). SMTP_USER/SMTP_PASS removed from credentials.json; replaced with GRAPH_TENANT_ID/CLIENT_ID/CLIENT_SECRET. +- **Vault stale backslash:** ad2.sops.yaml `credentials.password` contains `Paper123\!@#` (literal backslash). Must strip with `sed 's/\\//g'` at runtime. Cleanup pending. +- **Undocumented 2026-04-22 changes:** import.js, notify.js, upload-to-api.js modified that date with no session log. Changes appear stable but details unknown. +- **7B datasheet formatting:** SCM7B has ~830K records with specs loaded but needs a 7B-specific formatter layout. Not yet implemented. +- **SCM5B49 spec file empty:** 177000-15 (SCM5B49-05) needs empty spec file from John Lehman. Blocking one Quatronix datasheet. +- **New product lines not yet integrated:** MAQ20 (XLS format), PWRM10 (XLS), 10D (JSON, expected ~May 2026), DSCMHV — all need different parsers. +- **Diagnostic scripts on AD2:** `C:\Shares\testdatadb\database\_*.js` (~20 files from 2026-04-15 session) — safe to delete. +- **Service runs as INTRANET\svc_testdatadb** — credentials.json must be ACL'd to include Read + Traverse for this account (fixed 2026-04-15 but verify after changes). +- **psql not in PATH on AD2 for SSH sessions** — use Node.js pg module inline instead. +- **AD2 SSH port 22 intermittent** — sshd briefly unreachable for 5-15 min windows; ports 3000/3389/5985 stay up. Not a GuruRMM issue — likely network-layer (AV scan). Not caused by sshd crash. + +### QuickBASIC STR$() Formatting Quirk + +QuickBASIC's `STR$()` on a SINGLE emits two formats depending on magnitude: +- **Scientific with trailing test-status digit** (98.4%): `"PASS-7.005501E-033"` — trailing digit is test status code (2 or 3), not part of the value +- **Plain decimal, no trailing digit** (1.6%): `"PASS .01599373"` or `"PASS-.00499773"` + +Both formats encode percent units. Regex must try scientific first, plain decimal as fallback. Patch applied 2026-04-12. + +--- + +## Build & Deploy + +### Deploying Code to AD2 + +```bash +# From projects/dataforth-dos/deploy/ or datasheet-pipeline/implementation/ +python deploy-to-ad2.py + +# What it does: +# 1. Fetches AD2 password from vault (30s timeout, fails loud) +# 2. Connects via paramiko SFTP to 192.168.0.6:22 +# 3. Creates .bak-YYYYMMDD timestamped backups of existing files +# 4. Uploads modified files from implementation/ +# 5. Restarts testdatadb service via SSH exec_command +# 6. Verifies API responds 200 OK on port 3000 +``` + +Manual connection: +```bash +AD2_PASS=$(bash D:/vault/scripts/vault.sh get-field clients/dataforth/ad2.sops.yaml credentials.password | sed 's/\\//g') +ssh sysadmin@192.168.0.6 +``` + +### Checking Service Health + +```powershell +# On AD2 via SSH +Get-Service testdatadb # should show Running +Get-Service postgresql-18 # should show Running +``` + +```bash +# From workstation (needs VPN) +curl http://192.168.0.6:3000/api/stats +# Returns: {"totalRecords":469009,...} +``` + +### Scheduled Task Reference + +- Task name: `DataforthTestDatasheetUploader` +- Schedule: Daily 02:30 AM +- Runs as: SYSTEM +- Script: `C:\ProgramData\dataforth-uploader\run-pipeline.ps1` +- Logs: `C:\ProgramData\dataforth-uploader\logs\pipeline-YYYYMMDD.log` (60-day retention) + +--- + +## Active State + +**As of 2026-05-12 — pipeline is healthy.** See `projects/dataforth-dos/CONTEXT.md` for live detail. + +**Pending:** +- Enable SMTP AUTH for sysadmin@dataforth.com (AJ to do in Exchange Admin Center) OR confirm Graph API email path works (deployed 2026-05-12, not yet confirmed for live pipeline run) +- After email confirmed: add jlehman@dataforth.com to TO list in notify.js and run-pipeline.ps1 +- Clean diagnostic `_*.js` files from AD2 +- Fix vault stale backslash in ad2.sops.yaml +- Implement 7B datasheet formatter +- Integrate MAQ20/PWRM10/10D new product lines +- Investigate undocumented 2026-04-22 session changes + +--- + +## History Highlights + +| Date | Event | +|---|---| +| 2026-01-19 | DEPLOY.BAT added to Sync-FromNAS root-level sync. | +| 2026-01-20 | **DOS Update System production-ready.** 9 BAT files fixed (XCOPY /D param errors, STARTNET path), 39 deployments, pilot machine TS-4R. | +| 2026-03-11 | TestDataDB investigation — max test_date stuck at 2026-01-19; parser mtime issue. | +| 2026-03-12–16 | Import.js fixes, sync script improvements, Sync-FromNAS issues resolved. | +| 2026-03-27 | **Security incident — DF-JOEL2 compromised** (see Security Incident History section). MFA deployed. | +| 2026-03-27–29 | **Pipeline rebuilt** after 2025 crypto attack. New Node.js pipeline replaces DFWDS.exe + TestDataSheetUploader. Spec parser (1470 models), exact-match formatter, auto-export. 72/73 Quatronix datasheets generated. Root cause: CTONWTXT.BAT not called in AUTOEXEC v4.1 since 2026-03-12. | +| 2026-04-11 | Discovery session — SCMVAS/SCMHVAS research. hvin.dat decoded. Decision: accuracy-only template, no hvin.dat lookup. | +| 2026-04-12 | **SCMVAS/SCMHVAS pipeline extension deployed.** vaslog.js parser, accuracy-only formatter. 27,503 records backfilled (438 stragglers from QB STR$() quirk — patched same day). 434 Engineering-Tested .txt imported. Commit `0dd3d82`. | +| 2026-04-12 | TestDataDB PostgreSQL migration verified complete (2.89M records). SQLite archived. | +| 2026-04-13 | Hoffman API architecture finalized — client_credentials grant `dataforth.onprem.sync`. | +| 2026-04-14 | **DFWDS logic ported to dfwds-process.js (Node).** 897 staged datasheets drained. 803 new records created on Hoffman. End-to-end pipeline working. | +| 2026-04-15 | **Major release:** DB dedup (2.89M→469K unique SNs), FAIL→PASS retest rule, For_Web filesystem dependency eliminated (render in-memory), 170,984 records bulk-pushed to Hoffman. Dashboard pink tint, push buttons, bulk push, website status filter. PostgreSQL backup `test_records_dedup_bak_20260415` retained. | +| 2026-04-22 | Undocumented changes to import.js, notify.js, upload-to-api.js. No session log. | +| 2026-05-12 | **Email notifications implemented** — nodemailer then replaced with Graph API (Mail.Send granted to Claude-Code-M365 app). PS double-quote stripping issue discovered and worked around. SMTP_USER/PASS removed from credentials.json; GRAPH_TENANT_ID/CLIENT_ID/CLIENT_SECRET added. Pipeline fully documented in TEST-DATASHEET-PROCESS.md. | + +--- + +## Backlinks + +- [[clients/dataforth]] — Client article; this project runs on Dataforth's AD2 server +- [[systems/jupiter]] — Neptune Exchange physically housed at Dataforth D2; D2TESTNAS bridges Tailscale routing