azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9dd241626f51451eb6abf0b74e0ea561b6d074fb
claudetools/wiki/clients/dataforth.md
Howard Enos 941de499c2 sync: auto-sync from HOWARD-HOME at 2026-07-14 19:13:30 
Author: Howard Enos
Machine: HOWARD-HOME
Timestamp: 2026-07-14 19:13:30
2026-07-14 19:14:01 -07:00

60 KiB
Raw Blame History

type, name, display_name, last_compiled, compiled_by, sources, backlinks
type name display_name last_compiled compiled_by sources backlinks
client dataforth Dataforth Corporation 2026-07-14 HOWARD-HOME/claude-main
clients/dataforth/docs/overview.md
clients/dataforth/docs/active-directory.md
clients/dataforth/docs/workstations.md
clients/dataforth/docs/manufacturing.md
clients/dataforth/docs/billing-log.md
clients/dataforth/docs/SYNC_SCRIPT_UPDATE_SUMMARY.md
clients/dataforth/docs/incident-2026-03-27-abuse-report-virtuo.md
clients/dataforth/docs/incident-2026-03-27-abuse-report-connectwise.md
clients/dataforth/docs/cloud/m365.md
clients/dataforth/docs/issues/log.md
clients/dataforth/docs/network/topology.md
clients/dataforth/docs/network/vlans.md
clients/dataforth/docs/network/firewall.md
clients/dataforth/docs/rmm/rmm.md
clients/dataforth/docs/security/antivirus.md
clients/dataforth/docs/security/backup.md
clients/dataforth/docs/servers/ad1.md
clients/dataforth/docs/servers/ad2.md
clients/dataforth/docs/servers/d2testnas.md
clients/dataforth/docs/servers/df-hyperv-b.md
clients/dataforth/docs/servers/files-d1.md
clients/dataforth/docs/servers/sage-sql.md
clients/dataforth/docs/projects/shares-permissions/roadmap.md
clients/dataforth/docs/projects/shares-permissions/current-state-2026-06-10.md
clients/dataforth/docs/projects/shares-permissions/acl-audit-detail-2026-06-10.md
clients/dataforth/docs/projects/shares-permissions/discovery-email-draft.md
clients/dataforth/docs/projects/shares-permissions/target-structure-draft-2026-06-22.md
clients/dataforth/docs/aoi-xp-vlan-backup-runbook.md
clients/dataforth/session-logs/2026-03-23-galactic-advisors-report.md
clients/dataforth/session-logs/2026-03-27-security-incident-mfa-datasheets.md
clients/dataforth/session-logs/SESSION-SUMMARY.md
clients/dataforth/session-logs/MEMORY.md
clients/dataforth/session-logs/2026-04-12-session.md
clients/dataforth/session-logs/2026-04-13-session.md
clients/dataforth/session-logs/2026-04-14-session.md
clients/dataforth/session-logs/2026-04-23-session.md
clients/dataforth/session-logs/2026-05-03-session.md
clients/dataforth/session-logs/2026-05-04-lobby-phone-vlan-fix.md
clients/dataforth/session-logs/2026-05-06-session.md
clients/dataforth/session-logs/2026-05-12-session.md
clients/dataforth/session-logs/2026-06-01-aoi-xp-vlan-share.md
clients/dataforth/session-logs/2026-06-01-cbell-m365-bobbi-outlook.md
clients/dataforth/session-logs/2026-06-02-session.md
clients/dataforth/session-logs/2026-06-04-session.md
clients/dataforth/session-logs/project_ad2_context.md
clients/dataforth/session-logs/project_pipeline_rebuilt.md
clients/dataforth/session-logs/project_test_datasheet_pipeline.md
clients/dataforth/session-logs/project_new_product_lines.md
clients/dataforth/migration-gap-diff-RESUME.md
clients/dataforth/CLAUDE.dataforth.md
projects/dataforth-dos/CONTEXT.md
session-logs/2026-06-05-session.md
session-logs/2026-06/2026-06-09-mike-dataforth-freepbx-safesite-forensics.md
session-logs/2026-06/2026-06-18-mike-testdatadb-render-and-security-app.md
session-logs/2026-06/2026-06-29-mike-dataforth-nwtoc-pst-deletion-scope-birthbio-corruption.md
session-logs/2026-07/2026-07-13-howard-reliant-phones-dataforth-rdp-signing.md
.claude/memory/project_dataforth_incident_2026-03-27.md
.claude/memory/project_datasheet_pipeline.md
.claude/memory/project_neptune_sbr_email_routing.md
.claude/memory/reference_dataforth_contact.md
.claude/memory/reference_neptune_access_d2testnas.md
.claude/memory/feedback_d2testnas_ssh.md
.claude/memory/infra_office_network.md
.claude/memory/project_dataforth.md
.claude/memory/project_dataforth_history.md
.claude/memory/project_ad2_dataforth_fork.md
.claude/memory/ad2-ssh-mtu-blackhole.md
.claude/memory/ad2-comms-via-sync-only.md
clients/dataforth/session-logs/2026-06/2026-06-23-mike-pbx-no-inbound-calls-fix.md
clients/dataforth/session-logs/2026-06/2026-06-25-howard-dforth-ship-tdr-bsod.md
clients/dataforth/session-logs/2026-07/2026-07-01-mike-dataforth-test-data-chain-audit.md
clients/dataforth/session-logs/2026-07/2026-07-04-howard-mydata-tpsys-smt-controller-access.md
clients/dataforth/docs/audits/2026-07-01-test-data-chain-audit-AD2.md
clients/dataforth/session-logs/2026-07/2026-07-05-mike-dos-spec-fix-telemetry-udm.md
clients/dataforth/session-logs/2026-07/2026-07-06-mike-sage-sql-peter-session-reset.md
clients/dataforth/session-logs/2026-07/2026-07-06-winter-delfina-lopez-new-hire-account.md
clients/dataforth/session-logs/2026-07/2026-07-10-mike-haubner-ltobey-address-invalid.md
clients/dataforth/session-logs/2026-07/2026-07-14-howard-sage-printing-guru-user.md
clients/dataforth/session-logs/2026-07/2026-07-14-howard-pbx-inbound-dnat-flush-fix.md
.claude/memory/reference_rmm_spawn_headless_claude.md
projects/dataforth-dos
systems/jupiter

Dataforth Corporation

Signal conditioning / data acquisition manufacturer in Tucson, AZ. Long-standing ACG client. Active managed relationship — monthly prepaid block. Notable for 64 MS-DOS 6.22 test stations, a major security incident in March 2026, an ongoing test datasheet pipeline modernization project (DOS spec distribution fixed fleet-wide 2026-07-05), an incomplete 2025 post-ransomware recovery restore that silently dropped files across multiple shares (active audit underway), a shares/permissions remediation project (Phase 1 still pending client input), a legacy Fedora Core 3 SMT line controller discovered and recovered in July 2026, and Sage Pro 200 ERP running as an RDS RemoteApp on SAGE-SQL with RDP-redirected printing.

Profile

  • Contract type: Prepaid hour block (monthly replenishment invoice $2,098.87)
  • Hours remaining: 24.75 hrs as of 2026-07-14 (live Syncro GET /customers/578095 before billing)
  • Syncro customer ID: 578095
  • Syncro managed assets: 50
  • Billing rate: All invoices $0.00 — hours drawn from prepaid block
  • Invoice CC: jantar@dataforth.com

Key contacts:

Name Username Role Email
Dan Center dcenter Operations (primary IT contact) dcenter@dataforth.com
John Lehman jlehman Engineering, QB code, test specs jlehman@dataforth.com
Peter Iliya pIliya Applications Engineer pIliya@dataforth.com
Georg Haubner ghaubner Engineering; D: drive on HGHAUBNER has pre-ransomware backup of all DF shares ghaubner@dataforth.com
Kevin Wackerly kwackerly IT/Admin, handles calibration@ account kwackerly@dataforth.com
Logan Tobey ltobey Support/Sales ltobey@dataforth.com
Ben Wadzinski bwadzinski Engineering —
Lee Payne lpayne Engineering —
Theresa Dean tdean Admin tdean@dataforth.com
Joel Lohr jlohr RETIRED 2026-03-31 — account kept enabled; inbox rule forwards ntirety.com notifications to mike@azcomputerguru.com jlohr@dataforth.com
Ken Hoffman khoffman / oemdata TestDataSheetUploader author (external); owns Dataforth product API —
Delfina Lopez dlopez New hire (account created 2026-07-06, OU=Azure_Users) DLopez@dataforth.com
Winter Williams — Dataforth contact for Syncro cleanup requests —
  • External distributor: Ginger (gy@quatronix-cn.com) — Quatronix China; receives datasheets
  • Address: 3331 E. Hemisphere Loop, Tucson, AZ 85706 | (520) 741-1404

Open Syncro tickets (as of 2026-07-14):

Ticket Subject Status Date
#32505 Microsoft Access/Log-in request recent hire Customer Reply 2026-07-06
#32489 DOS test stations not pulling updated spec files (NWTOC v5.0 regression) Customer Reply 2026-06-29
#32441 Test datasheet generation defects + Phytec recovery New 2026-06-17
#32399 10 x Flex Mini Switch installation New 2026-06-09

Syncro Asset Reconciliation (2026-06-02)

Pulled 78 assets for customer_id 578095. Syncro managed count 50 (live, 2026-07-14). Fleet-wide Syncro agent break ~2025-10-06 silenced ~half the fleet while machines stayed online in ScreenConnect.

Bucket Count
KEEP 20
SAVE + FLAG (alive in SC/BD, Syncro agent broken — reinstall) 21
REMOVE (dead in all three systems) 28
VERIFY (servers, no agent anywhere) 9

Governing rule: machine is saved if online within 150 days in ANY of Syncro, ScreenConnect, or Bitdefender.

API delete not available — DELETE /customer_assets/{id} returns HTML 404 for current integration token. All deletions via Syncro GUI only.

Coord todo tree (assigned Howard, parent 103c48ad-7b31-4967-9388-065a91888e7c): delete 28 dead assets in GUI, decide 9 VERIFY servers, reinstall Syncro agent on 21 SAVE+FLAG machines, switch to metered billing once clean.

2026-07-08 GPS-RMM Reconciliation

GPS billing = 49 workstation + 3 server = 52 (matches RMM 52) per Winter/Mike.

Infrastructure

Servers & Services

Host IP Role OS Notes
AD1 192.168.0.27 Primary DC, DNS, FSMO roles, Engineering share Windows Server 2016 C:\ at 90% capacity (C:\Engineering = 787 GB) — critical risk. Image plan (Image2025) + Files plan (NBF, daily 2 AM, 180-day retention, C:\Engineering + C:\Shares\ITSvc). GuruRMM agent bf7bc5ee-4167-4a62-912a-c88b11a5943d.
AD2 192.168.0.6 Secondary DC, TestDataDB service host, NAS mirror, WebShare Windows Server 2019 (10.0.17763) Node.js TestDataDB on :3000. Wiped by 2025 crypto attack — rebuilt. Windows Firewall disabled (all profiles). Shares: C:\Shares\{c-drive,e-drive,webshare,test}. Old D:\c-drive volume gone — D: = mounted Windows install ISO. MSP360 agent C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe; storage account ACG-Dataforth. GuruRMM agent re-enrolled 2026-07-05: new id 4b9fe588-2a86-4855-912e-d90cf5f5e3e4 (old cfa93bb6-… stale). No shadow copies. Runs ClaudeTools on ad2 branch; coord-API isolated — comms via git sync only.
FILES-D1 192.168.0.189 File server Windows Server 2016 Shares: E:\Shares\{sales,archive}. staff share MISSING (see Patterns). GuruRMM agent 8566a19d-49a9-4f8b-9c6c-012cc934484b.
SAGE-SQL 192.168.0.153 Sage ERP (S:), RDS Session Host / Connection Broker / Web Access Windows Server 2016 Hosts Sage Pro 200 ERP 2010 (S:\sbt\sbtw.exe, Visual FoxPro). ~10 concurrent RDP users. RDS licensing grace period reset 2026-05-06. TSGateway disabled (not externally exposed). Self-signed cert CN=sage-sql.intranet.dataforth.com. Bitdefender GravityZone managed AV. Share: C:\sage. GuruRMM agent 120ba7bf-8544-48a0-98a1-40ed5cdd3e1f. Sage printers = RDP-redirected (Easy Print) only — native Brother/HP drivers NOT installed server-side (event 1111s); every session default = Microsoft Print to PDF. Sage app credential vault: clients/dataforth/sage.
3CX 192.168.0.125 Phone system (possibly inactive) — Last logon Oct 2025. Production phones live on VLAN 100 under Sangoma/FreePBX — 3CX role likely superseded.
DF-HYPERV-B 192.168.0.123 Hyper-V hypervisor Windows Server 2025 GuruRMM enrolled. Newest server in environment. VM inventory not captured.
DF-SVR-D2-Sync — (role TBD) — GuruRMM enrolled.
ENG-DEV-SERVER 192.168.0.126 Engineering dev server Windows 11 Pro GuruRMM enrolled.
D2TESTNAS 192.168.0.9 SMB1 bridge for DOS test stations + AOI XP backup; rsync source/target for the test-data pipeline Debian 13 (trixie), Samba 4.22.6 Repurposed Netgear ReadyNAS. SMB1 enabled globally (CORE..SMB3, NTLMv1) — required for DOS 6.22 stations. rsync daemon port 873 (module test, user rsync, hosts allow 192.168.0.0/24 + 172.16.0.0/12). SSH: root@192.168.0.9. Tailscale: subnet router, advertises 192.168.100.0/24 ONLY (not 192.168.0.0/24). Shares: test/datasheets/snapshots (guest; hosts deny 192.168.1.175), aoibackup (XP-only). Jump host for UDM SSH (direct-tcpip to 192.168.0.254). 2026-07-01 audit: root SSH key auth from AD2 is broken (publickey denied) — no impact on live rsync-daemon sync, but dormant SCP fallback script would fail.
ESXi hosts 192.168.0.122, 192.168.0.124 VMware ESXi hypervisors ESXi —
UDM Firewall 192.168.0.254 Perimeter firewall/router UniFi OS 5.1.15 MAC d0:21:f9:6c:11:02; also responds on 192.168.0.1. SSH: azcomputerguru@192.168.0.254 (2FA push) or root (keyboard-interactive password, vault clients/dataforth/udm.sops.yaml). Boot scripts /data/on_boot.d/: 10-neptune-snat.sh, 20-acg-network-isolation.sh, 30-freepbx-sip-forward.sh. [WARNING] UniFi Protect, Access, and Talk were uninstalled 2026-07-05 (were consuming ~1 GB RAM on a 4 GB device; Dataforth uses FreePBX, none were in use). UniFi Protect 7.1.83, Access 4.2.29, Talk 5.2.7 removed. C2 IPs blocked via iptables (NOT permanent — add to UniFi UI).
PBX (Sangoma FreePBX) 192.168.100.2 VoIP PBX — production phones on 192.168.100.0/24 Sangoma FreePBX 17 / Asterisk 22.5.2, Debian 12 FirstDigital PJSIP trunk; SBC 66.7.123.215:5060 (Sonus), match 66.7.123.0/24; IP-auth. qualify_frequency=0 (FD SBC ignores OPTIONS — do NOT revert). TFTP provisioning for Cisco SPA502G phones. Extensions 201–343. SSH: sangoma@192.168.100.2. sudo -S with password grants root (older "sudo not authorized" note was wrong — corrected 2026-07-14). Vault: clients/dataforth/pbx.sops.yaml. [WARNING] Re-apply PJSip.class.php line-504 patch after any fwconsole ma updateall. Durable fix applied 2026-07-14: native UniFi UI Port Forwarding policy "FreePBX-SIP" (WAN1 udp/5060, source-limited 66.7.123.0/24 → 192.168.100.2:5060) — provision-persistent, supersedes the on_boot.d script as primary; script retained as idempotent backup.
MYDATA TPSys SMT Controller (myserver) 192.168.1.1 (VLAN 2, SSH-confirmed 2026-07-04) MYDATA/Mycronic TPSys SMT pick-and-place line controller Fedora Core 3 "Heidelberg" (Nov 2004), kernel 2.6.16.20, glibc ~2.3.5, bash 3.00, LILO bootloader, SysV init Discovered + root recovered 2026-07-04. On VLAN 2 "mydata" (192.168.1.0/24). [WARNING] 192.168.1.1 collides with VLAN 2 gateway address — reconcile whether this box IS the gateway or there is an IP conflict with the UDM. TPSys UI runs under X (autologin, .profile → startx on tty1, default runlevel 3 not 5); local PostgreSQL backs TPSys. Root password reset via LILO recovery and vaulted: clients/dataforth/mydata-smt.sops.yaml. ACG RSA key: clients/dataforth/mydata-smt-ssh-key.sops.yaml (FC3 OpenSSH 3.9 needs legacy -o KexAlgorithms=+diffie-hellman-group14-sha1 -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa -o MACs=+hmac-sha1 -o Ciphers=+aes128-cbc). [CRITICAL] root must NOT be passwordless AND tpsys must remain in wheel with pam_wheel.so trust use_uid uncommented in /etc/pam.d/su — TPSys launcher /home/tpsys/bin/go escalates via su -c with no tty; a root password without the wheel trust breaks all escalation → X comes up empty (fvwm2 only, no TPSys UI). DR backup (2026-07-04): partition table + MBR/LILO + raw dd of OS partitions + /home tar + pg_dumpall, ~640 MB, SHA256-verified. On-site copy: D2TESTNAS:/root/dr-backups/mydata-tpsys/2026-07-04/. Offsite: AD1 C:\Shares\ITSvc\DR\mydata-tpsys\2026-07-04\ (inside AD1's MSP360 Files plan → B2 ACG-Dataforth, nightly 2 AM). GuruRMM agent cannot run on this box (glibc ~2.3.5 <2.17, kernel 2.6.16 <2.6.32, SysV init, no systemd).

Neptune Exchange (ACG infrastructure, physically at Dataforth D2):

  • neptune.acghosting.com | internal 172.16.3.11 | external inbound 67.206.163.124 / outbound 67.206.163.122
  • Exchange Server 2016, ACG-hosted mail server. Physically colocated at Dataforth D2 — NOT on ACG office LAN despite 172.16.x.x IP.
  • Access requires routing through D2TESTNAS (192.168.0.9).
  • SNAT rule: /data/on_boot.d/10-neptune-snat.sh on UDM forces Neptune outbound to .124.
  • Vault: clients/dataforth/neptune-exchange.sops.yaml
  • [WARNING] TODO: Resubnet Dataforth UDM to a non-overlapping range to permanently fix Neptune routing.

RMM-Spawned Claude on AD2

Proven capability (2026-07-01): a headless claude -p can be launched on AD2 through its GuruRMM agent (context:user_session, detached), writing a deliverable file + DONE.txt marker for a background poller. Bypasses AD2's coord-API isolation for read-only investigation. Must Remove-Item Env:\ANTHROPIC_API_KEY before invoking — a stale 108-char machine-level key shadows the working OAuth creds at C:\Users\sysadmin\.claude\.credentials.json. Reference: .claude/memory/reference_rmm_spawn_headless_claude.md.

Share -> Server -> Physical Path Map

Drive/Share Server Physical path Notes
Q: / c-drive AD2 C:\Shares\c-drive Old D:\c-drive is gone (D: = mounted install ISO)
T: / e-drive AD2 C:\Shares\e-drive DOS stations map T: to \\D2TESTNAS\test (separate)
X: / webshare AD2 C:\Shares\webshare DOS stations map X: to \\D2TESTNAS\datasheets (separate)
S: / sage SAGE-SQL C:\sage —
W: / sales FILES-D1 E:\Shares\sales —
Y: / archive FILES-D1 E:\Shares\archive —
B: / Engineering AD1 C:\Engineering —
B: / itsvc AD1 C:\Shares\ITSvc —
staff FILES-D1 — MISSING — share does not exist on FILES-D1

Workstations (summary)

Category Count OS Notable
Engineering ~12 Win 10/11 Pro HGHAUBNER (192.168.0.148) — Georg's PC; D: = full pre-attack backup of all 7 DF shares. GuruRMM agent 2aefe0d5-2357-4bdd-965a-abfccb4767a5. D1-PWRM for PWRM10 test.
Manufacturing/Assembly ~14 Win 10/11 Pro AS24, AS26 + various assembly/hi-pot stations
Office/Admin ~12 Win 10/11 Pro DF-GAGETRAK (192.168.0.102) — GAGEtrak calibration host. DF-JOEL2 (192.168.0.174) — compromised 2026-03-27, remediated. DESKTOP-U8PHS72 (192.168.0.48) — dlopez/kellynwackerly; local Brother DCP-L2550DW + HP LaserJet.
Shipping (part of Office/Admin) Win 10/11 Pro DFORTH-Ship — GuruRMM agent db17e069-2948-4cbc-97ea-1da721edcaf5, HP EliteDesk 800 G1 USDT. Recurring BSOD 0x116 VIDEO_TDR_FAILURE — see Patterns. Do not confuse with near-twin DForth-Shipp (95991b45-d843-4586-8275-9996d0d9ae17).
End-of-Life (Win 7) 3 Windows 7 Pro LABELPC (192.168.0.100), LABELPC2 (192.168.0.98), D2-RCVG-003 (192.168.0.47) — EOL, on network
AOI Optical Inspection (XP) 1 Windows XP WinXPBE-724667 @ 192.168.1.175 on VLAN 2. Backs up to \\192.168.0.9\aoibackup (SMB1, XP-only). EOL.
SMT Line Controller 1 Fedora Core 3 "Heidelberg" (2004) MYDATA TPSys (myserver), VLAN 2. Full detail in Servers table. No GuruRMM agent possible.
DOS Test Stations 64 MS-DOS 6.22 TS-1 through TS-30 + variants. Not domain-joined. SMB1 via D2TESTNAS. 14 remain active (confirmed 2026-07-05).

Email & Identity

  • M365 tenant: dataforth.com | Tenant ID: 7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584
  • Entra ID Sync: Yes — Azure AD Connect on AD1 (ADSync service). Synced OUs: OU=SyncedUsers and OU=Azure_Users. CompanyUsers OU does NOT sync.
  • M365 licenses: 50x Business Premium (40 used as of 2026-07-06), 19x Exchange Online Plan 1 (5 used), 5x SPB (4 used). Note: O365_BUSINESS_PREMIUM SKU (f245ecc8) is the legacy part number for M365 Business Standard, distinct from the 5-seat true Business Premium (SPB).
  • SMTP settings: smtp.office365.com, port 587, STARTTLS — use sysadmin@dataforth.com
  • SMTP AUTH status: Tenant-level not disabled; per-mailbox varies. calibration@dataforth.com SmtpClientAuthentication=true (re-enabled 2026-04-23). sysadmin@dataforth.com SMTP AUTH blocked by Exchange Online default — testdatadb uses Graph API for email (Mail.Send permission granted to Claude-Code-M365 app 2026-05-12).
  • Mail security stack (layered):
    1. INKY PhishFence — transport rule B859327F-3FBD-4BE7-A47A-97D02F1558A7, StopProcessingRules=true. Use inbox rules for per-mailbox routing, NOT transport rules.
    2. Mailprotector CloudFilter — outbound delivery gateway (dataforth-com.outbound.emailservice.io, 52.3.213.180). Active outbound connector "Outbound-Mailprotector". If a message shows "Delivered" in outbound trace but never arrives, check Mailprotector (/mailprotector skill).
  • DKIM: Both selectors published. Rotated 2026-05-12; cutover to selector2 on 2026-05-16.
  • DNS Host: ntirety.com — Dataforth's public DNS zone managed through ntirety's portal. DNS change requests go to ntirety, not a registrar panel. Joel Lohr's account retained to receive ntirety.com notifications (inbox rule → mike@azcomputerguru.com).
  • AutoForwarding: blocked by default (tenant outbound spam policy). If per-user forwarding needed, create scoped HostedOutboundSpamFilterPolicy with AutoForwardingMode=On.
  • MFA: 3 Conditional Access policies enforced 2026-04-04:
    • "ACG - Require MFA for All Users" — skip from office IP 67.206.163.122
    • "ACG - Block Foreign Sign-Ins" — US-only; MFA-Travel-Bypass group for exceptions
    • "ACG - Block Legacy Authentication"
  • Named locations: Dataforth Office - Tucson (67.206.163.122/32, trusted), Allowed Countries - US Only
  • MFA-Excluded-BreakGlass group: Brian Faires, Dataforth Calibration, Dataforth Notifications, Endcap, Tablet 01
  • Get-MessageTrace deprecated Sept 2025: Use Get-MessageTraceV2 and Get-MessageTraceDetailV2.

Network

  • Domain: intranet.dataforth.com | Forest/Domain Level: Windows Server 2016
  • ISP: fdtnet.net | Public IP: 67.206.163.122 (outbound), 67.206.163.124 (Neptune inbound)
  • Firewall/Router: UniFi Dream Machine Pro at 192.168.0.254 (also 192.168.0.1), UniFi OS 5.1.15
  • Network layout:
    • Main LAN: 192.168.0.0/24 (flat — servers, workstations, DOS stations)
    • Voice/PBX: VLAN 100 (192.168.100.0/24) — production phones. PBX at .2/.196.
    • VLAN 2 "mydata" (192.168.1.0/24): SMT production-line network; gateway 192.168.1.1. Members: WinXPBE-724667 (.175), goldstar19, DESKTOP-FT0T4MK, My9-PC, MYDATA TPSys (myserver, .1), + 3 unnamed industrial MACs (00:90:fb:80:f0:c6, 00:80:79:05:23:f2, 00:80:79:04:47:e7). Inter-VLAN routing mydata → main LAN currently OPEN.
  • VPN: OpenVPN for ACG remote access. Client subnet 192.168.6.x (GURU-5070 gets 192.168.6.2). [WARNING] GURU-5070 OpenVPN adapter "Local Area Connection" (ifIndex 12) MTU must be 1400 — default 1500 causes PMTU blackhole (tunnel path MTU ~1424; bulk SSH/SCP silently drops). Verify/re-apply: Set-NetIPInterface -InterfaceIndex 12 -AddressFamily IPv4 -NlMtuBytes 1400. Permanent fix: mssfix 1360 server-side.
  • Drive mappings (GPO): B: (\ad1\itsvc), Q: (\ad2\c-drive), S: (\SAGE-SQL\sage), T: (\ad2\e-drive), W: (\files-d1\sales), X: (\ad2\webshare), Y: (\files-d1\archive). DOS stations: T: (\D2TESTNAS\test), X: (\D2TESTNAS\datasheets).

GuruRMM Enrollment

  • Site name: Dataforth D1 | Site ID: 3a2f6866-26cd-452c-9806-a8df21475c3c
  • Site API key: vault clients/dataforth/... [check vault]
  • Fleet size: 45+ agents enrolled; Syncro managed count 50 as of 2026-07-14

[WARNING] GuruRMM enrollment workaround: WebSocket auth in ws/mod.rs does not validate enrolled_agents.agent_key_hash. New agent installs must overwrite registry AgentKey with the site API key and restart service. See Gitea issue #8.

[WARNING] rmm-api.azcomputerguru.com must be grey-clouded (DNS-only, not Cloudflare-proxied) — proxy blocks WebSocket. Gitea issue #9.

Known enrolled agents:

Host Agent ID Notes
AD1 bf7bc5ee-4167-4a62-912a-c88b11a5943d Enrolled 2026-06-04
AD2 4b9fe588-2a86-4855-912e-d90cf5f5e3e4 Re-enrolled 2026-07-05 (old cfa93bb6-… stale)
FILES-D1 8566a19d-49a9-4f8b-9c6c-012cc934484b Enrolled 2026-06-04
SAGE-SQL 120ba7bf-8544-48a0-98a1-40ed5cdd3e1f Enrolled 2026-06-04
DF-HYPERV-B (see RMM dashboard) Enrolled 2026-06-04
DF-SVR-D2-Sync (see RMM dashboard) Enrolled 2026-06-04
ENG-DEV-SERVER (see RMM dashboard) Enrolled 2026-06-04
DF-GAGETRAK 7626d82c-0736-47a6-8bc6-68e39859caed Enrolled 2026-04-23 (auth workaround applied)
HGHAUBNER 2aefe0d5-2357-4bdd-965a-abfccb4767a5 Georg's PC; pre-attack backup on D:
DFORTH-Ship db17e069-2948-4cbc-97ea-1da721edcaf5 Recurring TDR BSOD — see Patterns
DESKTOP-U8PHS72 25a75932-31cf-4b33-b94b-f4c404830877 dlopez/kellynwackerly workstation
(additional agents) — Full list in GuruRMM dashboard

Cannot be enrolled:

Host Reason
MYDATA TPSys (myserver) Fedora Core 3: glibc ~2.3.5 (<2.17), kernel 2.6.16 (<2.6.32), SysV init — three hard blockers. Plan agentless monitoring (ICMP/TCP or SSH heartbeat from D2TESTNAS).

Backup Architecture

  • MSP360 ("ACG-Online Backup", cbb.exe): Storage account ACG-Dataforth (account ID 0b49ca5e-…).
  • AD2: AD2 Image image plan + Files plan (180-day retention, NBF, daily 2 AM, C:\Shares tree; GFS off, synthetic full, compression, fast-NTFS). No shadow copies on AD2.
  • AD1: Image2025 image plan + Files plan (NBF, daily 2 AM, 180-day retention, covers C:\Engineering + C:\Shares\ITSvc; created 2026-06-05). Both image and file plans now match AD2.
  • Pre-attack backup (offline): HGHAUBNER D: drive holds a full pre-attack snapshot of all 7 mapped DF shares. The only recovery source predating the 2025 attack. Access via GuruRMM user_session on HGHAUBNER. Cross-machine writes use existing GPO-mapped drives only.
  • Historical file-level backup: NBF bunch faad5a67 in ACG-Dataforth storage — restore points 8/29–9/29/2025, archived at old D:\c-drive\... path (pre-migration layout).
  • WizTree backup CSV (2026-06-04): Full-drive export of HGHAUBNER's D: at AD2 C:\ClaudeTools\clients\dataforth\WizTree_20260604184904.zip (sensitive — keep off shares).
  • MYDATA TPSys DR (2026-07-04): hybrid image on D2TESTNAS + offsite via AD1 MSP360 Files plan → B2 ACG-Dataforth. See Servers table. TODO: full offline dd/Clonezilla image in a maintenance window.

Key Applications

Application Host URL/Port Notes
TestDataDB AD2 http://192.168.0.6:3000 Node.js + Express, PostgreSQL 18.3. Internal LAN only. UI redesigned 2026-06-18. 475,553 test_records as of 2026-07-01 audit; HTTP API uploader is the live web-delivery path.
Sage Pro 200 ERP 2010 SAGE-SQL RDS RemoteApp / S:\sbt\sbtw.exe Visual FoxPro-based. Users RDP to SAGE-SQL and run Sage there — printers = RDP-redirected (Easy Print). Native drivers not on server; default printer is Microsoft Print to PDF. Credential vault: clients/dataforth/sage (username: guru).
GageTrak DF-GAGETRAK (192.168.0.102) — Calibration tracking. Sends email via calibration@dataforth.com (SMTP AUTH re-enabled 2026-04-23). GuruRMM enrolled.
Dataforth Product API Hoffman's servers https://www.dataforth.com/api/v1/TestReportDataFiles OAuth2 client_credentials. Vault: clients/dataforth/api-oauth.sops.yaml. Token URL: https://login.dataforth.com/connect/token.
QuickBASIC 4.5 ATE 64 DOS stations T:\ (\D2TESTNAS\test) Automated test equipment. 1,470+ product model specs. 14 stations remain active (2026-07-05). Spec distribution pipeline fixed fleet-wide 2026-07-05 (NWTOC v5.1).
Power Monitor SPA Georg's dev / TBD — Vanilla-JS SPA for Dataforth power meters. Gateway architecture designed for PWM.dataforth.com demo. Parked pending Mike↔Georg conversation. clients/dataforth/power-monitor-demo/.
MYDATA TPSys MYDATA TPSys controller (myserver, VLAN 2) Local X UI / local PostgreSQL SMT pick-and-place line control. Discovered + documented 2026-07-04. See Servers table.

Access

Domain / Server Access

  • AD2 SSH: ssh sysadmin@192.168.0.6 (port 22) — vault: clients/dataforth/ad2.sops.yaml → credentials.password — strip backslash escape: sed 's/\\//g'. MTU-sensitive: GURU-5070 OpenVPN adapter ifIndex 12 must be MTU 1400.
  • AD1 SSH: ssh sysadmin@192.168.0.27 — vault: clients/dataforth/ad1.sops.yaml
  • D2TESTNAS SSH: ssh root@192.168.0.9 — vault: clients/dataforth/d2testnas.sops.yaml (password corrected 2026-07-14; was backslash-corrupted). Use root, NOT sysadmin. Tailscale: root@100.85.152.90. SSH key from GURU-5070 authorized; from BEAST: not authorized. From AD2, root SSH key auth is broken (publickey denied) — live rsync-daemon path unaffected.
  • D2TESTNAS aoibackup share: \\192.168.0.9\aoibackup — Samba user admin, hosts allow = 192.168.1.175 only. Creds: vault clients/dataforth/d2testnas.sops.yaml → credentials.smb.aoi-*.
  • UDM SSH: ssh azcomputerguru@192.168.0.254 (2FA push) or root (keyboard-interactive password; GURU-5070 id_ed25519 key also authorized). Jump via D2TESTNAS: paramiko direct-tcpip or ProxyJump. Vault: clients/dataforth/udm.sops.yaml (credentials.ssh_root_password = keyboard-interactive only — plink -pw and paramiko connect(password=) refused; use paramiko auth_interactive). From BEAST: no route to 192.168.0.x — jump through D2TESTNAS (advertises VLAN 100 only, not 192.168.0.x).
  • SAGE-SQL SSH: ssh sysadmin@192.168.0.153 — SSH key at C:\ProgramData\ssh\administrators_authorized_keys on SAGE-SQL. Times out from BEAST (no VPN route); use GuruRMM instead.
  • PBX SSH: sangoma@192.168.100.2 — vault: clients/dataforth/pbx.sops.yaml. sudo -S with same password grants root (note corrected 2026-07-14 — prior "sudo not authorized" entry was wrong).
  • WinRM (AD2/AD1): port 5985 — pywinrm with NTLM, user INTRANET\sysadmin
  • HGHAUBNER: No SSH. Reached via GuruRMM agent 2aefe0d5. Cross-machine file writes use existing GPO-mapped drives only.
  • All server passwords: vault (individual clients/dataforth/<host>.sops.yaml per server)

MYDATA TPSys SMT Controller

  • Root password: vault clients/dataforth/mydata-smt.sops.yaml. ACG RSA key: vault clients/dataforth/mydata-smt-ssh-key.sops.yaml (requires legacy SSH options).
  • Recovery if locked out again: LILO boot: → linux init=/bin/bash rw → passwordless root shell (bypasses sulogin). Then mount -o remount,rw / + passwd root. Reboot: reboot -f or echo b > /proc/sysrq-trigger.
  • DO NOT blank root, remove tpsys from wheel, or re-comment the pam_wheel.so trust line — any of these kills TPSys on next boot (all su escalations fail, X comes up empty).
  • Accounts: root, tpsys (app user, in wheel), tpspool (spool), postgres (uid 500, local DB).

M365 / Entra

  • M365 admin: sysadmin@dataforth.com — vault: clients/dataforth/m365.sops.yaml
  • Tenant ID: 7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584
  • Claude-Code-M365 Entra App: App ID 7a8c0b2e-57fb-4d79-9b5a-4b88d21b1f29, secret expires 2027-12-22 — vault: clients/dataforth/m365.sops.yaml → credentials.entra-app
  • MSP remediation app suite: MSP tenant ce61461e-81a0-4c84-bb4a-7b354a9a356d. Old single app fabb3421/Claude-MSP-Access DELETED 2026-06-14 — do not use. ComputerGuru tiered apps (Exchange Operator b43e7342, etc.) all consented 2026-04-23. Vault: msp-tools/computerguru-*.sops.yaml.
  • Contacts.Read not held by the investigator app in Dataforth tenant (ErrorAccessDenied, confirmed 2026-07-10). Exchange REST returns 401 (Exchange Admin role not assigned to investigator SP in this tenant).

MSP360 Managed Backup API

  • Vault: msp-tools/msp360-api.sops.yaml
  • cbb.exe path on AD2: C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe

Dataforth Product API (Hoffman)

  • Vault: clients/dataforth/api-oauth.sops.yaml
  • Token URL: https://login.dataforth.com/connect/token | Grant: client_credentials | Client ID: dataforth.onprem.sync | Scope: dataforth.web | Token TTL: 1 hour
  • Endpoints: GET /api/v1/TestReportDataFiles/{serial}, /bulk, /stats

ESXi / Hypervisors

  • ESXi-122: 192.168.0.122 — vault: clients/dataforth/esxi-122.sops.yaml
  • ESXi-124: 192.168.0.124 — vault: clients/dataforth/esxi-124.sops.yaml

Patterns & Known Issues

RDS / SAGE-SQL / Sage Printing

  • Sage Pro 200 ERP 2010 is an RDS RemoteApp only — users RDP to SAGE-SQL (192.168.0.153, Session Host) and run S:\sbt\sbtw.exe there. Sage is NOT installed on workstations.
  • Printers = RDP-redirected (Easy Print) — native Brother/HP drivers are NOT installed on SAGE-SQL (event 1111 "driver unknown"). Every session default = Microsoft Print to PDF. If a user sees no printers in Sage, first check for stale disconnected RDP sessions on SAGE-SQL.
  • Stale session printing fix (dlopez, 2026-07-14): user had two stale sessions (one active since prior day, one disconnected); fresh reconnects kept re-entering the stale sessions with zero printers. Fix: logoff <id> /v for all stale sessions via GuruRMM; new session brought printer redirection up correctly. Note: logoff exits nonzero on success — use query user output as truth, not exit code.
  • Registry change for RDP behavior was applied prior to 2026-07-14 (before that session); fresh logon required to take effect. Watch-item: if Sage printing complaints recur with the Easy Print queues, install native Brother DCP-L2550DW + HP Universal PCL 6 drivers on SAGE-SQL.
  • RDS licensing: Grace period reset 2026-05-06 by deleting GracePeriod registry key. Purchase RDS CALs (Per User mode, LicensingType=4).
  • TSGateway: Disabled on SAGE-SQL (server not externally exposed). Do NOT re-enable without reason.
  • SSL cert: Self-signed, CN=sage-sql.intranet.dataforth.com. Non-domain machines must import to Trusted Root + Trusted Publishers manually.
  • Bitdefender GravityZone managed AV on SAGE-SQL — may block PowerShell execution; temporary disable may be needed for admin work.
  • Sage credential vault: clients/dataforth/sage (username: guru). Password reset by Dataforth staff 2026-07-14; verified working. Never inline the password.

Active Directory

  • No custom security groups — only default Windows groups. Service accounts in OU=ServiceAccounts.
  • ClaudeTools-ReadOnly AD account — purpose unclear. Investigate.
  • Ken Hoffman has two accounts (khoffman + oemdata) — not consolidated.
  • jlohr account retained — post-retirement (2026-03-31); kept enabled for ntirety.com notifications. Inbox rule forwards to mike@azcomputerguru.com. Do NOT disable.
  • Entra sync scope: OU=SyncedUsers and OU=Azure_Users sync to Entra. CompanyUsers OU does NOT sync. 38 stale TS-* test station accounts cleaned from Entra 2026-03-27.
  • New hire pattern (dlopez, 2026-07-06): create on-prem in OU=Azure_Users, add to RDWeb-Users, trigger ADSync delta. ADSync service runs on AD1 itself.
  • SKU disambiguation: O365_BUSINESS_PREMIUM (f245ecc8) = M365 Business Standard (legacy part number), NOT true Business Premium (SPB). This is what standard user accounts hold.

Voice / Phones / FreePBX

  • Production phones VLAN: 192.168.100.0/24. PBX at .2/.196. All production phones live here.
  • VLAN 2 (192.168.1.0/24) is the mydata/SMT VLAN — do NOT confuse with voice. Phones landing on 192.168.1.x cannot reach PBX.
  • D1-Server-Room port 1: Controls lobby drop → must stay on VLAN 100. Reverted to default once before (2026-05-04 incident).
  • FirstDigital trunk — qualify_frequency=0: FD's Sonus SBC ignores SIP OPTIONS keepalives. Non-zero qualify causes trunk to go Unavailable and blocks all INVITEs. Do NOT revert.
  • PJSip.class.php line 504 patch must be re-applied after any fwconsole ma updateall — wiped by FreePBX updates. Backup before each update.
  • Do NOT port-forward the RTP range (10000-20000) on the UDM. Static RTP DNAT causes a conntrack collision with the PBX's outbound RTP — inbound works but outbound audio dies. SIP 5060 forward only, source-locked to 66.7.123.0/24.
  • Inbound SIP DNAT (recurring failure, now durably fixed): Previously relied only on /data/on_boot.d/30-freepbx-sip-forward.sh, which is flushed by any UniFi controller provision mid-uptime (not just reboots). Outage 2026-07-13 11:05 AM → 2026-07-14 09:39 AM fixed by: (1) re-running the on_boot.d script (immediate fix), (2) creating a native UniFi UI Port Forwarding policy "FreePBX-SIP" (WAN1 udp/5060, source 66.7.123.0/24 → 192.168.100.2:5060) that survives provisions. Coord to-do 45572ee1 (durable fix) is now closed. on_boot.d script retained as an idempotent backup. Diagnostic: iptables -t nat -S | grep -iE '5060|192.168.100.2' on UDM — empty means rule is gone.
  • sangoma sudo clarification: sangoma is in the sudo group and sudo -S with the password DOES grant root (prior "sudo not authorized" note was wrong — corrected 2026-07-14). asterisk -rx still denied for sangoma (socket is owner-writable only).
  • Prior outages: 2026-06-08 (#32392), 2026-06-23 (#32450), 2026-07-14 (#32553).

Exchange Online / Email

  • INKY PhishFence StopProcessingRules: Kills all subsequent transport rules. Use inbox rules for per-mailbox forwarding, NOT transport rules.
  • Mailprotector CloudFilter: Outbound delivery goes through Mailprotector. If "Delivered" per outbound trace but message never arrives, check Mailprotector (/mailprotector skill).
  • AutoForwarding blocked by default — create scoped HostedOutboundSpamFilterPolicy with AutoForwardingMode=On if needed per sender.
  • Autocomplete stale-cache trap (Haubner→Tobey, 2026-07-10): Outlook displays the correct SMTP address from cache but routes via a stored EX/LegacyExchangeDN that no longer resolves. Fix: user deletes the stale autocomplete entry in Outlook and re-picks from GAL. Remote cache-clear more disruptive (file locked while Outlook is open, nukes all cached names).
  • Contacts.Read / Exchange REST gaps: The investigator app does not have Contacts.Read in Dataforth (ErrorAccessDenied). Exchange REST returns 401 (Exchange Admin role not assigned to investigator SP in this tenant). Use GuruRMM for workstation-level inspection when these APIs are blocked.

GuruRMM Agent Deployment

  • WebSocket auth bug (Issue #8): enrolled_agents.agent_key_hash not validated. Workaround: overwrite registry HKLM:\SOFTWARE\GuruRMM\AgentKey with site API key, restart service.
  • rmm-api.azcomputerguru.com must be grey-clouded — Cloudflare proxy blocks WebSocket. Gitea Issue #9.
  • AD2 re-enrolled 2026-07-05: new agent id 4b9fe588-2a86-4855-912e-d90cf5f5e3e4. Old id cfa93bb6-… is stale — do not dispatch to it.
  • No RMM agent possible on FC3-class legacy appliances: confirmed on MYDATA TPSys (glibc ~2.3.5, kernel 2.6.16, SysV init — three hard blockers). Plan agentless monitoring (ICMP/TCP probe or SSH heartbeat from D2TESTNAS).
  • SYSTEM token cross-machine writes: GuruRMM user_session commands use an impersonated token with no network credentials. Fresh UNC paths fail with Access Denied. Use existing GPO-mapped drives (Q:, T:, etc.) for cross-machine writes — proven on HGHAUBNER 2026-06-04.
  • Legacy Red-Hat-family recovery pattern: if locked out, LILO boot: → linux init=/bin/bash rw → passwordless root shell (bypasses sulogin on Red-Hat family).
  • RMM dispatch JSON quoting: inline JSON with \\ escapes fails the API's JSON parse. Use ps-encoded.sh for PowerShell dispatches. query session exits nonzero on success — use stdout content as truth.

AD2 SSH / VPN MTU

  • PMTU blackhole on GURU-5070 → AD2 SSH: OpenVPN adapter ifIndex 12 defaults MTU 1500; tunnel path MTU is ~1424. Bulk TCP segments silently dropped; interactive commands pass. Presents as "flaky VPN" or "random lockouts."
  • Fix (applied 2026-06-18, registry-persistent): Set-NetIPInterface -InterfaceIndex 12 -AddressFamily IPv4 -NlMtuBytes 1400 via GURU-5070 RMM agent. May reset on OpenVPN reconnect — re-apply if stalling resumes.
  • Durable fix: mssfix 1360 server-side on the Dataforth OpenVPN server (fleet-wide).
  • AD2 is NOT the target for SSH diagnosis when SSH is the failing channel — use RMM.

AD2 Branch / Coordination

  • AD2 operates on the ad2 git branch. Fork = main + thin Dataforth-specific commits. Never edit shared fleet files on ad2. Dataforth context: clients/dataforth/CLAUDE.dataforth.md.
  • AD2 is coord-API isolated: 172.16.3.30 unreachable from Dataforth LAN. All inter-session coordination via git sync: committed handoff docs + ## Note for <user> blocks. Do NOT use the coord skill for AD2.
  • sync.sh push gotcha: always fails at Phase 4 on ad2 branch (pushes main). Force-push manually: git push --force-with-lease origin ad2.
  • Stale machine-level ANTHROPIC_API_KEY on AD2 (108 chars, invalid) shadows working OAuth creds. Must Remove-Item Env:\ANTHROPIC_API_KEY before invoking claude -p.
  • AD2 RMM side channel: GuruRMM agent (4b9fe588-…, current) can spawn a headless claude -p for live read-only ground-truth investigation (proven 2026-07-01). Unset the stale API key first.

Post-Ransomware Recovery Restore (2025) — Incomplete File Migration

  • The 10/1/2025 recovery restore was incomplete. Restore plan 10/1/2025 (~3.4M files) migrated each share from old D:\<share> to C:\Shares\... on AD2 and dropped files in the process. Proven case: SP1366 MAQ20 — each PRINTOUTS FOR MANUFACTURING folder for revisions E–H received only one file when the backup contained ~6 per revision.
  • Scope unknown. A full migration-gap audit is underway (WizTree both sides). Review-only — no automatic restore (some deletions were intentional; HGHAUBNER backup is additive-only and includes personal files).
  • AD2 D: drive is gone. Old D:\c-drive data volume repurposed as mounted Windows install ISO. Historical file-level backup (bunch faad5a67) archived data under D:\c-drive\... (pre-migration path) — reconcile paths accordingly.

Shares ACL State — All Open to All Staff

  • All 8 business shares grant access to every employee (Everyone/Domain Users Full or Modify on all). No department-based security groups. Payroll, OSHA, Purchase Orders, Accounting/QuickBooks, Sage financials fully readable/writable by all domain users.
  • Remediation project in progress (Shares & Permissions, started 2026-06-10). Phase 0 (discovery) complete. Phase 1 (client input) pending discovery email to Dan Center. Do not apply ACL changes until after client sign-off.
  • Special shares excluded from department-model remediation: test (DOS/SMB1 guest — leave open), webshare (preserve svc_testdatadb:Full), ITSvc (Domain Computers needs Read), Sage app data path.
  • Phase 2 target-state strawman (drafted 2026-06-22): Company\Departments\ tree with SG-<Resource>-<RW|RO> groups, Restricted\ branch with broken inheritance, ABE on. Option A (keep current drive letters, reorganize folders within) recommended for least disruption. Full detail: clients/dataforth/docs/projects/shares-permissions/.
  • Six items gate Phase 2 sign-off: confirmed department list; department → share access matrix; named access for sensitive data; department rosters; legacy cleanup approval; Engineering destination volume (AD1 C: ~90% full blocks any ENGR restructure).

DOS Test Station Data Pipeline

  • DOS spec distribution fixed fleet-wide 2026-07-05 (Syncro #32489 root-caused 2026-07-01, fix deployed 2026-07-05). All 14 active stations confirmed carrying current 5BMAIN.DAT (83,200, 06-26-26).
  • Root cause (F1, resolved): deployed NWTOC.BAT v5.0 copied only *.BAT and *.EXE — zero .DAT files — by design (its own changelog said so). No version of NWTOC ever distributed shared COMMON master specs; v5.1 added that copy.
  • F2 (resolved): COPY /Y is not a valid MS-DOS 6.22 switch. Mike confirmed all stations are genuine 6.22 — /Y caused NWTOC to copy nothing fleet-wide since 2026-03-16. Fixed in v5.1 (plain COPY).
  • NWTOC v5.1 deployed 2026-07-05 (plain COPY, adds one-way .DAT pull from COMMON/ProdSW into C:\ATE\<type>DATA, ATTRIB -R guard, CRLF-forced). Validated on TS-3L, promoted fleet-wide. v5.0 backed up to NWTOC50.BAK. CTONWTXT updated to v2.4 (drops /Y). CTONW at v5.2 (flat-file telemetry, no network MD) then v5.3 pending (CTONW v5.3 dispatch went to stale AD2 id and needs redoing).
  • Tier-A telemetry deployed: SYSTAT.BAT (plain paused status page) and flat-file telemetry to T:\STATUS\%MACHINE%.{CFG,AUT,SPV,MEM} — 14 stations collected cleanly 2026-07-05.
  • Rollout is per-station opt-in (broken v5.0 cannot bootstrap its own replacement — each station needs one manual COPY seed, then self-maintains).
  • Secondary cleanup items (not yet done): remove stray TS-21\ProdSW file (F3 — breaks AD2↔NAS rsync push every run); update testdatadb\specdata\ from live engineering masters (F4 — frozen 2026-03-27 snapshot); rotate/vault plaintext creds in scripts (F5); retire dead For_Web output and abandoned v1.x script drafts.
  • Full 2026-07-01 audit report: clients/dataforth/docs/audits/2026-07-01-test-data-chain-audit-AD2.md.

Hardware / Endpoint — Aging Fleet

  • DFORTH-Ship recurring TDR BSOD (0x116 VIDEO_TDR_FAILURE, diagnosed 2026-06-25): integrated Intel HD Graphics 4600, driver 20.19.15.5126 (Intel's final, 2020-01-20), on an 11.5-year-old HP EliteDesk 800 G1 USDT (BIOS 2014-12-10). Five minidumps 2025-11-03 through 2026-06-24, accelerating cadence. Mitigation: Edge hardware acceleration disabled via machine policy (HKLM\SOFTWARE\Policies\Microsoft\Edge\HardwareAccelerationModeEnabled = 0). No durable fix for integrated graphics — PC replacement is the real fix; thermal cleaning is a secondary measure. Verify Edge policy at edge://policy after restart. Do not confuse with near-twin DForth-Shipp (95991b45-…) — verify agent ID before acting.
  • MYDATA TPSys controller (~20 years old, FC3 Nov 2004): documented and DR-backed 2026-07-04 after being completely undocumented until discovered at the physical console. Worth a broader sweep for other undocumented devices on VLAN 2 — 3 unnamed industrial MACs already known but unresolved as of 2026-06-01.
  • End-of-Life machines on network: 3x Windows 7 (LABELPC, LABELPC2, D2-RCVG-003), 1x Windows XP (WinXPBE-724667/AOI). All EOL, all unpatched.

Security

  • C2 IP blocks are iptables only — do not survive UDM reboot. Must add 80.76.49.18 and 45.88.91.99 (AS399486 Virtuo, Montreal) to permanent UniFi UI block list.
  • AD1 disk 90% full — C:\Engineering = 787 GB of 1023 GB. Risk of replication failures. Blocks Engineering share restructure.
  • Windows Firewall disabled on AD2 (all profiles) — known risk, not remediated.
  • AD1/AD2 on Server 2016/2019 — approaching end of mainstream support.
  • Entra ID P2 not licensed — IdentityRiskyUser risk check returns 403 (no P2).
  • Plaintext credentials in Dataforth test-data-chain scripts (F5, 2026-07-01): rsync daemon password in Sync-FromNAS-rsync.ps1, Postgres testdatadb_app password in testdatadb\database\db.js — world-readable via the test SMB share. Rotation + vaulting recommended, not yet done.
  • UniFi Protect/Access/Talk removed 2026-07-05 — were consuming ~1 GB RAM on 4 GB UDM while unused (Dataforth uses FreePBX). All three uninstalled. UDM management plane briefly down during post-uninstall reboot; data plane (forwarding) remained live.

Syncro Asset Management

  • Fleet-wide Syncro agent break ~2025-10-06: ~half of Dataforth machines stopped reporting to Syncro while remaining online in ScreenConnect. Hard cutoff, not gradual attrition. Do NOT auto-remove machines frozen at that date without cross-checking ScreenConnect.
  • Bitdefender is NOT a liveness signal: Dataforth phasing off BD; 53 of 57 GravityZone endpoints in "Deleted" folder. Missing from BD = BD agent uninstalled.
  • API delete not available — all asset deletions via Syncro GUI (/customer_assets?customer_id=578095).

staff Share Missing

The staff network share is absent from FILES-D1 (only archive and sales exist). HGHAUBNER's backup includes a DF Staff folder, suggesting the share existed pre-attack. Not in scope for the current migration-gap diff — separate issue requiring investigation.

Active Work

As of 2026-07-14 (4 open Syncro tickets per live pull):

  • DOS Test Station Data Pipeline (Syncro #32489, active): Root cause confirmed and fix deployed fleet-wide 2026-07-05 (NWTOC v5.1). All 14 active stations confirmed with current specs. Ticket status is Customer Reply — awaiting Dataforth to confirm on their end. Pending: (1) promote CTONW v5.3 (dispatch to stale AD2 agent id needs redoing with new id 4b9fe588-…); (2) secondary cleanup items F3–F5 from the 2026-07-01 audit (stray TS-21 file, frozen specdata, plaintext creds in scripts); (3) close ticket once client confirms resolution.

  • Test Datasheet Pipeline (Syncro #32441, open):

    • Production outbound pipeline healthy — 475K+ records, DSCA33/45 recovery complete (1,452 certs published 2026-06-18 via Hoffman API).
    • Email notifications deployed (Graph API via sysadmin@dataforth.com).
    • 8B/5B/SCM render gap (~9,624 records unpublished) — 136 templates mined, wiring handed to AD2 (parked).
    • testdatadb\specdata\ frozen at 2026-03-27 snapshot — any spec limit changed after that date not reflected in server-side generated datasheets even though the outbound pipeline is healthy (F4).
    • 2 niche DSCA models (DSCA33-1948, DSCA45-1746) have no Hoffman original — cannot auto-publish.

  • Flex Mini Switch installation (Syncro #32399, open): 10 x Flex Mini Switch installation. No further detail captured in reviewed session logs.

  • New hire account/access (Syncro #32505, Customer Reply): Delfina Lopez account created 2026-07-06 in AD + M365. Ticket is in Customer Reply — follow up with Winter.

  • DFORTH-Ship BSOD (ongoing monitoring): Edge hardware-acceleration mitigation applied 2026-06-25; needs on-site Edge restart/reboot to take effect. Monitor for recurrence. If it bugchecks again, pull older dump signatures to confirm hardware fault trajectory. Schedule thermal cleaning. Recommend PC replacement as durable fix.

  • UDM inbound SIP DNAT (closed 2026-07-14): Durable fix applied — native UI Port Forwarding rule "FreePBX-SIP" created. Coord to-do 45572ee1 closed. Syncro ticket #32553 (0.5 hr, block debited 25.25 → 24.75 hrs). Monitor stability.

  • Shares & Permissions project (Phase 1 — BLOCKING, pending client input): Phase 0 complete 2026-06-10. Discovery email to Dan Center drafted (docs/projects/shares-permissions/discovery-email-draft.md) — not yet sent (recipients/sender not locked). Phase 1 blocked on client responses: department list, access matrix, sensitive-data rules, rosters. Phase 2 strawman drafted 2026-06-22. Next step: polish client HTML and send discovery email.

  • MYDATA TPSys SMT controller: Root recovered and documented 2026-07-04. Outstanding: (1) reconcile 192.168.1.1 IP vs VLAN 2 gateway conflict; (2) verify tpsys wheel-group + NOPASSWD sudoers change survived the reboot (id tpsys, sudo -l as tpsys); (3) stand up agentless monitoring (ICMP/TCP probe or SSH heartbeat from D2TESTNAS — inter-VLAN routing to mydata is open); (4) full offline dd/Clonezilla image in a maintenance window.

  • Migration-gap audit (parked): WizTree CSV of HGHAUBNER's pre-attack backup captured. WizTree on live servers deferred — no diff yet. Plan: run WizTree on AD2, FILES-D1, SAGE-SQL, AD1 → diff per share → clients/dataforth/migration-gap-catalog-2026-06-04.md. Full plan: clients/dataforth/migration-gap-diff-RESUME.md.

  • Syncro asset cleanup (with Howard): 28 confirmed-dead assets pending GUI deletion; 21 machines need Syncro agent reinstall; 9 servers in VERIFY bucket. Coord todo tree 103c48ad-… assigned to Howard. Switch to metered billing once clean.

  • AOI XP backup + isolation (ongoing): AOI XP @ 192.168.1.175 on VLAN 2; aoibackup SMB1 share locked to XP only. Optional EOL hardening pending: block XP → company LAN (except NAS 192.168.0.9) + Internet on UDM. Todo 37543f7f.

  • AD2 Claude capability updates (parked): AD2 needs syncro + coord commands, DF wiki read-write, Dataforth client data access. Coord API unreachable from Dataforth LAN — comms via git sync + RMM-spawn side channel.

  • Power Monitor SPA demo (parked): Georg Haubner's vanilla-JS power-meter SPA. Gateway spec at clients/dataforth/power-monitor-demo/GATEWAY-SPEC.md. Parked pending Mike↔Georg conversation.

  • MFA enrollment: 19 users not enrolled at April 4 enforcement date; current enrollment count unverified.

  • C2 IP blocks need permanence: iptables rules on UDM (80.76.49.18, 45.88.91.99) need to be added to permanent UniFi UI block list.

  • jlohr forwarding: ntirety.com inbox rule active as of 2026-05-12. Defunct transport rule pending cleanup.

  • RDS / SAGE-SQL: GPO cert distribution pending. RDS CALs purchase needed long-term.

  • Haubner → Tobey autocomplete fix (2026-07-10): Diagnosed as stale cached EX/LegacyExchangeDN in Outlook autocomplete. Recommended fix: Haubner deletes the Logan entry in Outlook and re-picks from GAL. Awaiting Mike's direction on delivery method (instructions to Haubner vs scheduled remote clear).

History Highlights

Date Event
2025 Crypto/ransomware attack — AD2 wiped and rebuilt, many files lost. Test datasheet pipeline broken.
2025-08-29 – 2025-09-29 MSP360 file-level backup (faad5a67) covering DF shares at old D:\c-drive\... path. Last snapshot before the recovery restore.
2025-10-01 – 2025-10-02 Post-ransomware recovery restore (Restore plan 10/1/2025, ~3.4M files) migrated shares from D:\<share> to C:\Shares\... on AD2. Restore was incomplete — files dropped in multiple folders.
~2025-10-06 Fleet-wide Syncro agent break — ~half of Dataforth machines freeze in Syncro while remaining online in ScreenConnect. Root cause unknown.
2026-01-19 DOS Update System built and deployed — NWTOC/CTONW/UPDATE/DEPLOY BAT files, 39 deployments.
2026-03-20 Galactic Advisors security assessment — AD1 C: at 90%, legacy SQL 2008 R2 client noted.
2026-03-27 Major security incident: DF-JOEL2 compromised via social engineering/ScreenConnect (attacker "Angel Raya", C2 on Virtuo hosting). M365 sign-in from Turkey. Full remediation. 3 CA policies deployed. MFA notice sent. IC3 filed (1c32ade367084be9acd548f23705736f).
2026-03-27–29 Test datasheet pipeline rebuilt — new Node.js pipeline replaces VB6 DFWDS + VB.NET uploader. 72/73 Quatronix datasheets generated.
2026-03-31 Joel Lohr retirement. Brian Faires mailbox converted to shared. 38 stale Entra TS-* accounts deleted.
2026-04-04 MFA CA policies enforced (switched from report-only).
2026-04-11–12 SCMVAS/SCMHVAS pipeline extension — 27,503 records backfilled, 434 Engineering-Tested .txt files imported.
2026-04-12 TestDataDB PostgreSQL migration verified (2.89M records). Hoffman API discovered.
2026-04-13 API architecture discussion with Hoffman — client_credentials grant confirmed.
2026-04-14 DFWDS logic ported to Node.js. 897 staged datasheets drained. 803 new records pushed to Hoffman API.
2026-04-15 Major release — DB dedup (2.89M→469K rows), FAIL→PASS retest rule, 170,984 records bulk-pushed to Hoffman. Dashboard upgrades.
2026-04-23 Full Dataforth tenant onboarded to all 5 ComputerGuru tiered apps. calibration@ SMTP AUTH fixed. DF-GAGETRAK GuruRMM enrolled. Syncro #32142 billed.
2026-05-03 jantar@dataforth.com darkweb breach check — no indicators of compromise. eM Client OAuth grant revoked.
2026-05-04 Howard onsite — lobby phone offline (VLAN misconfiguration on D1-Server-Room port 1 → fixed to VLAN 100).
2026-05-06 SAGE-SQL RDS issues resolved — grace period reset, SSL cert replaced, TSGateway disabled.
2026-05-12 Pipeline audit + email notifications implemented (Graph API). jlohr forwarding configured. DKIM keys rotated.
2026-06-01 AOI optical-inspection XP PC isolated onto VLAN 2 @ 192.168.1.175; aoibackup SMB1 share created on D2TESTNAS locked to the XP only. D2TESTNAS confirmed Debian 13 / Samba 4.22.6.
2026-06-01 Chauncey Bell (cbell) M365 verified; Bobbi's Outlook printing fixed. Ticket #32364 (0.5 hr onsite).
2026-06-02 Syncro asset reconciliation (78 assets): 20 keep / 21 save+flag / 28 remove / 9 verify.
2026-06-04 SP1366 MAQ20 manufacturing print recovery — 19/20 PDFs for revisions E–H restored from HGHAUBNER pre-attack backup. GuruRMM fleet grew 13 → 45 agents. WizTree backup-side CSV captured.
2026-06-05 AD1 Files backup plan created (cbb.exe, NBF, 180-day retention, C:\Engineering + C:\Shares\ITSvc).
2026-06-05 Mailprotector CloudFilter discovered as Dataforth's outbound delivery layer. Email from Georg held by Mailprotector; released manually. /mailprotector skill built.
2026-06-05 Georg Haubner's Power Monitor SPA analyzed. Gateway architecture designed for PWM.dataforth.com demo. Parked.
2026-06-08–09 Total Dataforth phone outage. Outbound failed (qualify_frequency → trunk Unavailable); inbound never worked (no SIP port-forward existed). Fixed: qualify_frequency=0 in pjsip DB; PJSip.class.php line 504 re-patched; 30-freepbx-sip-forward.sh added. Two-way audio verified. Syncro #32392, 1.0 hr emergency (×1.5 rate).
2026-06-10 Shares & Permissions Phase 0 complete. All 8 business shares open to all employees. Phase 1 pending.
2026-06-17 AD2 identity.json + Python 3.12.8 installed. CLAUDE.dataforth.md created.
2026-06-18 DSCA33/45 certs recovered via Hoffman API — 1,452 new certs published. parseRawData bug root-caused. AD2 SSH PMTU blackhole diagnosed and fixed (MTU 1400). TestDataDB UI redesigned. Syncro #32441.
2026-06-22 Shares & Permissions Phase 2 target-state strawman drafted — Company\Departments\…Restricted\… tree with `SG--<RW
2026-06-23 PBX no-inbound-calls emergency fixed. UDM SIP 5060 DNAT flushed by UniFi controller provision (not a reboot). Re-ran on_boot.d script; verified inbound restored. Vault pbx.sops.yaml password corrected. Syncro #32450, 1.5 hr emergency (×1.5 rate), block 31.5 → 30.0 hrs.
2026-06-24–25 DFORTH-Ship recurring BSOD diagnosed (0x116, Intel HD 4600, 11.5-yr-old EliteDesk). Edge hardware-acceleration mitigation applied. PC replacement recommended.
2026-07-01 Test-data-chain ground-truth audit via RMM-spawned headless Claude on AD2. Confirmed NWTOC v5.0 never copies .DAT files (F1); COPY /Y invalid on 6.22 (F2); three additional findings. Stale assumptions corrected (PostgreSQL 18, rsync-daemon sync path, HTTP API uploader is live web-delivery).
2026-07-04 MYDATA TPSys SMT controller (myserver, FC3/VLAN2) discovered + root recovered via LILO single-user; vaulted; RMM agent ruled out. DR backup taken and stored on-site + offsite.
2026-07-05 DOS spec distribution fixed fleet-wide. NWTOC v5.1 deployed (adds .DAT pull, plain COPY, no /Y). All 14 active DOS stations confirmed with current specs. Tier-A station telemetry added. UDM: UniFi Protect/Access/Talk uninstalled (memory savings). unifi.service restarted (0-clients issue — RabbitMQ/WebSocket memory starvation).
2026-07-06 Delfina Lopez (dlopez) new hire account created in AD (OU=Azure_Users) + M365 (Business Standard). Ticket #32505.
2026-07-06 SAGE-SQL session reset for piliya (Peter Iliya, disconnected session 13). Ticket #32506, no billing.
2026-07-08 GPS-RMM billing reconciliation: 49 workstation + 3 server = 52 (matches RMM).
2026-07-10 Haubner→Tobey "address invalid" diagnosed — stale autocomplete EX/LegacyExchangeDN on Haubner's Outlook. Awaiting fix delivery decision.
2026-07-13 – 2026-07-14 PBX inbound outage (2nd UDM SIP DNAT flush, 07-13 ~11:05 AM → 07-14 09:39 AM AZ). Fixed by re-running on_boot.d script; durable fix: native UI Port Forwarding policy "FreePBX-SIP" created (provision-persistent). Coord to-do 45572ee1 closed. Vault: UDM root password corrected (keyboard-interactive only); d2testnas password de-corrupted. Sangoma sudo-works note corrected. Syncro #32553, 0.5 hr, block 25.25 → 24.75 hrs.
2026-07-14 Sage printing fix (dlopez). Diagnosed: stale RDP sessions on SAGE-SQL (no redirected printers until logged off). Logged off stale sessions via GuruRMM; fresh RDP brought printer redirection up. Sage app credential (guru user) vaulted at clients/dataforth/sage. Syncro #32552, 0.5 hr, block 25.75 → 25.25 hrs (then 25.25 → 24.75 from #32553 same day; final live block = 24.75).

Backlinks

Reference in New Issue View Git Blame Copy Permalink