azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
85c8149495a6e5024acc8d5e19137ba5ff6af8ec
claudetools/wiki/clients/dataforth.md
Howard Enos 85c8149495 sync: auto-sync from HOWARD-HOME at 2026-07-04 12:00:16 
Author: Howard Enos
Machine: HOWARD-HOME
Timestamp: 2026-07-04 12:00:16
2026-07-04 12:00:44 -07:00

75 KiB
Raw Blame History

type, name, display_name, last_compiled, compiled_by, sources, backlinks
type name display_name last_compiled compiled_by sources backlinks
client dataforth Dataforth Corporation 2026-07-04 Howard-Home/claude-main
clients/dataforth/docs/overview.md
clients/dataforth/docs/active-directory.md
clients/dataforth/docs/workstations.md
clients/dataforth/docs/manufacturing.md
clients/dataforth/docs/billing-log.md
clients/dataforth/docs/SYNC_SCRIPT_UPDATE_SUMMARY.md
clients/dataforth/docs/incident-2026-03-27-abuse-report-virtuo.md
clients/dataforth/docs/incident-2026-03-27-abuse-report-connectwise.md
clients/dataforth/docs/cloud/m365.md
clients/dataforth/docs/issues/log.md
clients/dataforth/docs/network/topology.md
clients/dataforth/docs/network/vlans.md
clients/dataforth/docs/network/firewall.md
clients/dataforth/docs/rmm/rmm.md
clients/dataforth/docs/security/antivirus.md
clients/dataforth/docs/security/backup.md
clients/dataforth/docs/servers/ad1.md
clients/dataforth/docs/servers/ad2.md
clients/dataforth/docs/servers/d2testnas.md
clients/dataforth/docs/servers/df-hyperv-b.md
clients/dataforth/docs/servers/files-d1.md
clients/dataforth/docs/servers/sage-sql.md
clients/dataforth/docs/projects/shares-permissions/roadmap.md
clients/dataforth/docs/projects/shares-permissions/current-state-2026-06-10.md
clients/dataforth/docs/projects/shares-permissions/acl-audit-detail-2026-06-10.md
clients/dataforth/docs/projects/shares-permissions/discovery-email-draft.md
clients/dataforth/docs/projects/shares-permissions/target-structure-draft-2026-06-22.md
clients/dataforth/session-logs/2026-06/2026-06-23-howard-dataforth-share-plan-recovery.md
clients/dataforth/docs/aoi-xp-vlan-backup-runbook.md
clients/dataforth/session-logs/2026-03-23-galactic-advisors-report.md
clients/dataforth/session-logs/2026-03-27-security-incident-mfa-datasheets.md
clients/dataforth/session-logs/SESSION-SUMMARY.md
clients/dataforth/session-logs/MEMORY.md
clients/dataforth/session-logs/2026-04-12-session.md
clients/dataforth/session-logs/2026-04-13-session.md
clients/dataforth/session-logs/2026-04-14-session.md
clients/dataforth/session-logs/2026-04-23-session.md
clients/dataforth/session-logs/2026-05-03-session.md
clients/dataforth/session-logs/2026-05-04-lobby-phone-vlan-fix.md
clients/dataforth/session-logs/2026-05-06-session.md
clients/dataforth/session-logs/2026-05-12-session.md
clients/dataforth/session-logs/2026-06-01-aoi-xp-vlan-share.md
clients/dataforth/session-logs/2026-06-01-cbell-m365-bobbi-outlook.md
clients/dataforth/session-logs/2026-06-02-session.md
clients/dataforth/session-logs/2026-06-04-session.md
clients/dataforth/session-logs/project_ad2_context.md
clients/dataforth/session-logs/project_pipeline_rebuilt.md
clients/dataforth/session-logs/project_test_datasheet_pipeline.md
clients/dataforth/session-logs/project_new_product_lines.md
clients/dataforth/migration-gap-diff-RESUME.md
clients/dataforth/CLAUDE.dataforth.md
projects/dataforth-dos/CONTEXT.md
session-logs/2026-06-05-session.md
session-logs/2026-06/2026-06-09-mike-dataforth-freepbx-safesite-forensics.md
session-logs/2026-06/2026-06-18-mike-testdatadb-render-and-security-app.md
.claude/memory/project_dataforth_incident_2026-03-27.md
.claude/memory/project_datasheet_pipeline.md
.claude/memory/project_neptune_sbr_email_routing.md
.claude/memory/reference_dataforth_contact.md
.claude/memory/reference_neptune_access_d2testnas.md
.claude/memory/feedback_d2testnas_ssh.md
.claude/memory/infra_office_network.md
.claude/memory/project_dataforth.md
.claude/memory/project_dataforth_history.md
.claude/memory/project_ad2_dataforth_fork.md
.claude/memory/ad2-ssh-mtu-blackhole.md
.claude/memory/ad2-comms-via-sync-only.md
clients/dataforth/session-logs/2026-06/2026-06-23-mike-pbx-no-inbound-calls-fix.md
clients/dataforth/session-logs/2026-06/2026-06-25-howard-dforth-ship-tdr-bsod.md
clients/dataforth/session-logs/2026-07/2026-07-01-mike-dataforth-test-data-chain-audit.md
clients/dataforth/session-logs/2026-07/2026-07-04-howard-mydata-tpsys-smt-controller-access.md
clients/dataforth/docs/audits/2026-07-01-test-data-chain-audit-AD2.md
.claude/memory/reference_rmm_spawn_headless_claude.md
projects/dataforth-dos
systems/jupiter

Dataforth Corporation

Signal conditioning / data acquisition manufacturer in Tucson, AZ. Long-standing ACG client. Active managed relationship — monthly prepaid block. Notable for 64 MS-DOS 6.22 test stations, a major security incident in March 2026, an ongoing test datasheet pipeline modernization project, an incomplete 2025 post-ransomware recovery restore that silently dropped files across multiple shares (active audit underway), a shares/permissions remediation project (Phase 1 still pending client input), and — newly documented as of 2026-07-04 — a previously-undocumented legacy Linux SMT line controller (MYDATA TPSys, Fedora Core 3) discovered on the manufacturing VLAN.

Profile

  • Contract type: Prepaid hour block (monthly replenishment invoice $2,098.87)
  • Key contacts:
Name Username Role Email
Dan Center dcenter Operations (primary IT contact) dcenter@dataforth.com
John Lehman jlehman Engineering, QB code, test specs jlehman@dataforth.com
Peter Iliya pIliya Applications Engineer pIliya@dataforth.com
Georg Haubner ghaubner Engineering; D: drive on HGHAUBNER has pre-ransomware-attack backup of all DF shares ghaubner@dataforth.com
Kevin Wackerly kwackerly IT/Admin, handles calibration@ account kwackerly@dataforth.com
Logan Tobey ltobey Support/Sales ltobey@dataforth.com
Ben Wadzinski bwadzinski Engineering —
Lee Payne lpayne Engineering —
Theresa Dean tdean Admin tdean@dataforth.com
Joel Lohr jlohr RETIRED 2026-03-31 — account intentionally kept enabled; inbox rule forwards ntirety.com notifications to mike@azcomputerguru.com jlohr@dataforth.com
Ken Hoffman khoffman / oemdata TestDataSheetUploader author, external; also owns Dataforth product API —
Winter — Dataforth contact who requested Syncro asset cleanup 2026-06-02 —
  • External distributor: Ginger (gy@quatronix-cn.com) — Quatronix China; receives datasheets
  • Billing rate: Prepaid block; all invoices show $0.00 — hours drawn from block
  • Hours remaining: 30.0 hrs as of 2026-07-04 (live-check Syncro before billing — GET /customers/578095)
  • Syncro customer ID: 578095
  • Syncro managed assets: 50
  • Open Syncro tickets: 0 as of 2026-07-04
  • Invoice CC: jantar@dataforth.com

Infrastructure

Servers & Services

Host IP Role OS Notes
AD1 192.168.0.27 Primary DC, DNS, FSMO roles, Engineering share Windows Server 2016 C:\ at 90% capacity (C:\Engineering = 787 GB) — critical risk. FSMO roles (assumed all). GuruRMM agent bf7bc5ee-4167-4a62-912a-c88b11a5943d. Image plan (Image2025) + Files plan (NBF, daily 2 AM, 180-day retention — created 2026-06-05).
AD2 192.168.0.6 Secondary DC, TestDataDB service host, NAS mirror, WebShare Windows Server 2019 (10.0.17763) Hosts testdatadb Node.js service on :3000. Wiped by crypto attack 2025 — rebuilt. Windows Firewall disabled (all profiles). Shares: C:\Shares\{c-drive,e-drive,webshare,test}. Old D:\c-drive data volume is GONE — D: is now a mounted Windows install ISO. MSP360 agent at C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe; storage account ACG-Dataforth. GuruRMM agent cfa93bb6-0cdc-4d4e-a29e-1609cda6f047. No shadow copies. Runs ClaudeTools on ad2 branch (coord-API isolated; comms via git sync only). New (2026-07-01): its GuruRMM agent can also be used to spawn a headless claude -p for live read-only ground truth — see RMM-Spawned Claude below; console user sysadmin, claude.exe v2.1.181 at C:\Users\sysadmin\.local\bin\, node v20.10.0. OS build corrected 2026-07-01 (was previously logged as 2022; live audit confirmed Server 2019).
FILES-D1 192.168.0.189 File server Windows Server 2016 Shares: E:\Shares\{sales,archive}. GuruRMM agent 8566a19d-49a9-4f8b-9c6c-012cc934484b. NOTE: staff share is missing on FILES-D1 — separate issue.
SAGE-SQL 192.168.0.153 Sage ERP (S:), RDS Session Host/Connection Broker/Web Access Windows Server 2016 RDS licensing grace period was expired (reset 2026-05-06). TSGateway disabled (server not externally exposed). New self-signed RDS cert installed. Bitdefender GravityZone managed AV. Share: C:\sage. GuruRMM agent 120ba7bf-8544-48a0-98a1-40ed5cdd3e1f.
3CX 192.168.0.125 Phone system (possibly inactive) — Last logon Oct 2025. Production phones live on VLAN 100 under the Sangoma/FreePBX PBX — 3CX role likely superseded.
DF-HYPERV-B 192.168.0.123 Hyper-V hypervisor Windows Server 2025 GuruRMM enrolled. Newest server in environment. VM inventory not captured.
DF-SVR-D2-Sync — (role TBD) — GuruRMM enrolled
ENG-DEV-SERVER 192.168.0.126 Engineering dev server Windows 11 Pro GuruRMM enrolled
D2TESTNAS 192.168.0.9 SMB1 bridge for DOS test stations + AOI XP backup; Neptune Exchange colocation routing; rsync source/target for the test-data pipeline Debian 13 (trixie), Samba 4.22.6 Repurposed Netgear ReadyNAS. SMB1 enabled globally (CORE..SMB3, NTLMv1) — required for DOS 6.22 stations. rsync daemon on port 873 (module test, user rsync, hosts allow 192.168.0.0/24 + 172.16.0.0/12). SSH: root@192.168.0.9. Tailscale route for 172.16.0.0/22. Shares: test/datasheets/snapshots (guest; hosts deny 192.168.1.175), aoibackup (XP-only — see Access). Acts as jump host for UDM SSH (D2TESTNAS direct-tcpip channel to 192.168.0.254). 2026-07-01 audit: root SSH key auth from AD2 (root@192.168.0.9) is broken (publickey denied) — no operational impact since the live sync uses the rsync daemon, not SSH, but the dormant SCP fallback script would fail if re-enabled (F9).
ESXi hosts 192.168.0.122, 192.168.0.124 VMware ESXi hypervisors ESXi —
UDM Firewall 192.168.0.254 Perimeter firewall/router UniFi OS 5.1.15 MAC d0:21:f9:6c:11:02. Also responds on 192.168.0.1. SSH: azcomputerguru@192.168.0.254, root SSH key added 2026-06-08, 2FA push required. Vault: clients/dataforth/udm.sops.yaml. C2 IPs blocked via iptables (NOT permanent — need to add to UniFi UI). Boot scripts in /data/on_boot.d/: 10-neptune-snat.sh (Neptune outbound SNAT), 30-freepbx-sip-forward.sh (SIP DNAT, WAN UDP 5060 source-locked to 66.7.123.0/24 → 192.168.100.2; SIP-only — do NOT add RTP forward). [WARNING] Confirmed 2026-06-23: the SIP DNAT rule can be silently flushed by a UniFi controller provision/update, not only a reboot — the on_boot.d script only re-applies at boot, so a mid-uptime provision event leaves inbound calls dead until the script is manually re-run. Recommend adding a persistent UI port-forward rule as a belt-and-suspenders measure (still not done as of 2026-07-04).
PBX (Sangoma FreePBX) 192.168.100.2 VoIP PBX — production phones on 192.168.100.0/24 Sangoma FreePBX 17 / Asterisk 22.5.2, Debian 12 FirstDigital PJSIP trunk; SBC 66.7.123.215:5060 (Sonus), match 66.7.123.0/24; IP-auth (no registration). qualify_frequency=0 (FD SBC ignores OPTIONS — do NOT revert). TFTP provisioning for Cisco SPA502G phones. Extensions 201-343. SSH: sangoma@192.168.100.2 (ACG SSH key also authenticates). Vault: clients/dataforth/pbx.sops.yaml — password corrected 2026-06-23 (prior entry had a backslash-escaping corruption in the stored value; re-verify the vault entry byte-for-byte before use rather than assuming it is stale). [WARNING] Re-apply PJSip.class.php line-504 patch after any fwconsole ma updateall. NOTE: sangoma user is in the sudo group but sudo authorization on this box appears not actually granted — verify before assuming privileged ops will work via sudo.
MYDATA TPSys SMT Controller (myserver) 192.168.1.x (verify — exact IP unconfirmed) MYDATA/Mycronic TPSys pick-and-place SMT production-line controller Fedora Core 3 "Heidelberg" (Nov 2004), kernel 2.6.16.20, glibc ~2.3.5, bash 3.00, LILO bootloader, SysV init (no systemd) NEW, discovered 2026-07-04. On VLAN 2 "mydata" (192.168.1.0/24, gateway 192.168.1.1). TPSys operator UI runs under X (runlevel 5); local PostgreSQL (uid 500) backs TPSys. Accounts: root, tpsys (TPSys app user), tpspool (TPSys spool), postgres. No credential existed anywhere (vault or wiki) prior to this discovery — root password was RESET via physical-console LILO recovery (no prior password to lose) and is now vaulted at clients/dataforth/mydata-smt.sops.yaml (reference the vault path only; never the raw password). tpsys being added to the wheel group plus a scoped NOPASSWD sudoers entry for the app-launch command (directed, not yet verified as landed). GuruRMM agent CANNOT run on this box — see Patterns below.

Neptune Exchange (ACG infrastructure, physically at Dataforth D2):

  • neptune.acghosting.com | internal 172.16.3.11 | external inbound 67.206.163.124 / outbound 67.206.163.122
  • Exchange Server 2016, active ACG-hosted mail server for multiple clients
  • Physically colocated at Dataforth's D2 facility — NOT on ACG office LAN despite 172.16.x.x IP
  • Access requires routing through D2TESTNAS (192.168.0.9): Dataforth UDM has a 172.16.x.x subnet that overlaps ACG office LAN, making direct routing ambiguous
  • SNAT rule on Dataforth UDM at /data/on_boot.d/10-neptune-snat.sh should force Neptune outbound to use .124 (not always active — verify)
  • Vault: clients/dataforth/neptune-exchange.sops.yaml
  • [WARNING] TODO: Resubnet Dataforth UDM to a non-overlapping range to permanently fix Neptune routing

RMM-Spawned Claude on AD2

  • New capability proven 2026-07-01: a headless claude -p can be launched on AD2 through its GuruRMM agent (cfa93bb6-..., context:user_session, since sysadmin is logged into the console and the RMM elevated token works), detached (Start-Process ... -WindowStyle Hidden) so it survives the RMM command-timeout window, writing a deliverable file + DONE.txt marker that a background poller watches. This is a live read-only ground-truth channel that works despite AD2 being isolated from the ACG coord API (172.16.3.30 unreachable from Dataforth LAN) — it does NOT replace the git-sync handoff for anything that needs to write back to the shared repo.
  • Gotcha: a stale machine-level ANTHROPIC_API_KEY (108 chars, invalid) on AD2 shadows sysadmin's working OAuth credentials (C:\Users\sysadmin\.claude\.credentials.json). Must Remove-Item Env:\ANTHROPIC_API_KEY before invoking claude -p, or it fails with Invalid API key.
  • Reference: .claude/memory/reference_rmm_spawn_headless_claude.md; full pattern + transcript in clients/dataforth/session-logs/2026-07/2026-07-01-mike-dataforth-test-data-chain-audit.md.

Share -> Server -> Physical Path Map

Drive/Share Server Physical path Notes
Q: / c-drive AD2 C:\Shares\c-drive Old D:\c-drive is gone (D: = mounted install ISO)
T: / e-drive AD2 C:\Shares\e-drive On AD2 itself, T: is \\ad2\e-drive — NOT the NAS. DOS stations separately map their own T: to \\D2TESTNAS\test (see DOS Test Station Data Pipeline pattern).
X: / webshare AD2 C:\Shares\webshare On AD2, X: = \\ad2\webshare. DOS stations separately map their own X: to \\D2TESTNAS\datasheets.
S: / sage SAGE-SQL C:\sage —
W: / sales FILES-D1 E:\Shares\sales —
Y: / archive FILES-D1 E:\Shares\archive —
B: / Engineering AD1 C:\Engineering —
B: / itsvc AD1 C:\Shares\ITSvc —
staff FILES-D1 — MISSING — share does not exist on FILES-D1

Workstations (summary)

Category Count OS Notable
Engineering ~12 Win 10/11 Pro HGHAUBNER (192.168.0.148) — Georg's PC; D: = full pre-attack backup of all 7 DF shares (DF C-Drive, DF E-Drive, DF WebShare, DF Sage, DF Server Sales/Archive/Engineering, + personal). GuruRMM agent 2aefe0d5-2357-4bdd-965a-abfccb4767a5. D1-PWRM for PWRM10 test.
Manufacturing/Assembly ~14 Win 10/11 Pro AS24, AS26 + various assembly/hi-pot stations
Office/Admin ~12 Win 10/11 Pro DF-GAGETRAK (192.168.0.102) — GAGEtrak calibration host. DF-JOEL2 (192.168.0.174) — compromised 2026-03-27, remediated.
Shipping (part of Office/Admin count) Win 10/11 Pro DFORTH-Ship — GuruRMM agent db17e069-2948-4cbc-97ea-1da721edcaf5, HP EliteDesk 800 G1 USDT (BIOS 2014-12-10, ~11.5 yrs old). Recurring BSOD 0x116 VIDEO_TDR_FAILURE on integrated Intel HD Graphics 4600 — see Patterns. Do not confuse with near-twin host DForth-Shipp (95991b45-d843-4586-8275-9996d0d9ae17), a separate machine.
End-of-Life (Win 7) 3 Windows 7 Pro LABELPC (192.168.0.100), LABELPC2 (192.168.0.98), D2-RCVG-003 (192.168.0.47) — EOL, on network
AOI Optical Inspection (XP) 1 Windows XP WinXPBE-724667 @ 192.168.1.175 on VLAN 2 (mydata/SMT). Holds the AOI machine's external drive; backs up to \\192.168.0.9\aoibackup (SMB1, XP-only). EOL. See AOI runbook + 2026-06-01 session log.
SMT Line Controller (legacy Linux appliance) 1 Fedora Core 3 "Heidelberg" (2004) MYDATA TPSys (myserver), on VLAN 2 (mydata/SMT), ~20 years old. Full detail in Infrastructure Servers table above. No GuruRMM agent possible (glibc/kernel/init all below the agent's floor) — agentless monitoring planned.
DOS Test Stations 64 MS-DOS 6.22 TS-1 through TS-30 + variants (62 TS-* folders confirmed on NAS 2026-07-01). Not domain-joined. SMB1 via D2TESTNAS. See DOS Test Station Data Pipeline pattern for the 2026-07-01 audit findings.

Email & Identity

  • M365 tenant: dataforth.com | Tenant ID: 7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584
  • Entra ID Sync: Yes — Azure AD Connect. Synced OUs include OU=SyncedUsers and OU=Azure_Users (cbell confirmed in OU=Azure_Users and syncing, 2026-06-01) — the earlier "SyncedUsers only" note was incomplete.
  • M365 licenses: 50x Business Premium (39 used), 19x Exchange Online Plan 1 (5 used), 5x SPB (4 used)
  • SMTP settings: smtp.office365.com, port 587, STARTTLS — use sysadmin@dataforth.com
  • SMTP AUTH status: Tenant-level not disabled; per-mailbox varies. calibration@dataforth.com had SmtpClientAuthentication=true re-enabled 2026-04-23. sysadmin@dataforth.com SMTP AUTH is blocked by Exchange Online default — testdatadb uses Graph API for email (Mail.Send permission granted to Claude-Code-M365 app 2026-05-12).
  • Mail security stack (layered):
    1. INKY PhishFence — active transport rule B859327F-3FBD-4BE7-A47A-97D02F1558A7 fires first (StopProcessingRules=true). Use inbox rules for per-user mail routing, NOT transport rules.
    2. Mailprotector CloudFilter — outbound delivery gateway (dataforth-com.outbound.emailservice.io, 52.3.213.180). Active outbound connector "Outbound-Mailprotector" (recipientDomains *). Mail may be held here. If a message shows "Delivered" in Dataforth outbound trace but never arrives, check Mailprotector (/mailprotector skill). Discovered 2026-06-05 when ghaubner email was held by "INKY - Annotation - Recipient Not Group Member" transport rule.
  • DKIM: Both selector1 and selector2 published. Rotated 2026-05-12; cutover to selector2 on 2026-05-16.
    • selector1._domainkey.dataforth.com → selector1-dataforth-com._domainkey.dataforthcom.onmicrosoft.com
    • selector2._domainkey.dataforth.com → selector2-dataforth-com._domainkey.dataforthcom.onmicrosoft.com
  • DNS Host: ntirety.com — Dataforth's public DNS zone managed through ntirety's portal (not a standard registrar). DNS change requests go to ntirety, not a domain control panel. Joel Lohr's account retained to receive ntirety.com infrastructure notifications (inbox rule → mike@azcomputerguru.com).
  • AutoForwarding blocked by default (tenant outbound spam policy). If per-user forwarding needed, create scoped HostedOutboundSpamFilterPolicy for that sender with AutoForwardingMode=On.
  • MFA: 3 Conditional Access policies created 2026-03-27 (initially report-only; enforced 2026-04-04):
    • "ACG - Require MFA for All Users" — skip from office IP 67.206.163.122
    • "ACG - Block Foreign Sign-Ins" — US-only; MFA-Travel-Bypass group for exceptions
    • "ACG - Block Legacy Authentication"
  • Named locations: Dataforth Office - Tucson (67.206.163.122/32, trusted), Allowed Countries - US Only
  • MFA-Excluded-BreakGlass group: Brian Faires, Dataforth Calibration, Dataforth Notifications, Endcap, Tablet 01
  • MFA enrollment (as of 2026-03-27): 19/38 ready, 19 needed setup — deadline April 4, 2026

Network

  • Domain: intranet.dataforth.com | Forest/Domain Level: Windows Server 2016
  • ISP: fdtnet.net | Public IP: 67.206.163.122 (outbound), 67.206.163.124 (Neptune inbound)
  • Firewall/Router: UniFi Dream Machine Pro at 192.168.0.254 (also 192.168.0.1), UniFi OS 5.1.15
  • Network: Flat (no VLANs on main LAN — 192.168.0.0/24). Voice/PBX VLAN: 192.168.100.0/24 — production phones live here. VLAN 2 "mydata" (192.168.1.0/24) = SMT production-line network (gateway 192.168.1.1); members on the D2-SMT Switch (USW Enterprise 8) + D2-Breakroom port 12. Supersedes the earlier note that 192.168.1.0/24 was an unused UDM default voice VLAN — it is in active use by SMT. Inter-VLAN routing from mydata → main LAN is currently OPEN.
    • mydata members (2026-06-01): WinXPBE-724667 (AOI XP, .175), goldstar19, DESKTOP-FT0T4MK, My9-PC, + 3 unnamed industrial/SMT devices (MAC 00:90:fb:80:f0:c6, 00:80:79:05:23:f2, 00:80:79:04:47:e7).
    • mydata addition (2026-07-04): MYDATA TPSys controller (myserver) confirmed present on this VLAN — hostname distinct from the named members above; exact IP still to confirm (verify). Likely corresponds to one of the previously "unnamed industrial" devices or is a device not yet captured in the member list; reconcile on next VLAN sweep.
  • VPN: OpenVPN for ACG remote access. Client subnet 192.168.6.x (GURU-5070 gets 192.168.6.2). [WARNING] GURU-5070 OpenVPN adapter "Local Area Connection" (ifIndex 12) MTU must be set to 1400 — default 1500 causes PMTU blackhole (tunnel path MTU ~1424; bulk SSH/SCP silently drops). Verify/re-apply: Set-NetIPInterface -InterfaceIndex 12 -AddressFamily IPv4 -NlMtuBytes 1400. Permanent fix: add mssfix 1360 server-side on the Dataforth OpenVPN server.
  • Drive mappings (GPO): B: (\ad1\itsvc), Q: (\ad2\c-drive), S: (\SAGE-SQL\sage), T: (\ad2\e-drive), W: (\files-d1\sales), X: (\ad2\webshare), Y: (\files-d1\archive). DOS test stations: T: (\D2TESTNAS\test), X: (\D2TESTNAS\datasheets)

GuruRMM Enrollment

  • Site name: Dataforth D1 | Site ID: 3a2f6866-26cd-452c-9806-a8df21475c3c
  • Site API key: vault clients/dataforth/... [check vault for current entry]
  • Fleet size: 45 agents enrolled as of 2026-06-04; Syncro managed count 50 as of 2026-07-04
  • [WARNING] GuruRMM enrollment workaround: WebSocket auth in ws/mod.rs does not validate enrolled_agents.agent_key_hash. New agent installs must overwrite registry AgentKey with the site API key (not the enrollment AgentKey) and restart service. See Gitea issue #8.
  • [WARNING] Agent floor confirmed 2026-07-04 (MYDATA TPSys case): the Linux installer (agent/scripts/install.sh) requires glibc ~2.17+, kernel >=2.6.32, and a systemd host. Legacy Linux appliances below that floor (e.g. Fedora Core 3, SysV init) cannot run the agent — plan agentless monitoring (ICMP/TCP probe, SSH heartbeat from a reachable host) for such boxes instead of attempting install.

Known enrolled agents:

Host Agent ID Notes
DF-GAGETRAK 7626d82c-0736-47a6-8bc6-68e39859caed Enrolled 2026-04-23 (auth workaround applied)
HGHAUBNER 2aefe0d5-2357-4bdd-965a-abfccb4767a5 Georg's PC; pre-attack backup on D:
AD2 cfa93bb6-0cdc-4d4e-a29e-1609cda6f047 Enrolled 2026-06-04; also used 2026-07-01 to spawn headless Claude for the test-data-chain audit
AD1 bf7bc5ee-4167-4a62-912a-c88b11a5943d Enrolled 2026-06-04
FILES-D1 8566a19d-49a9-4f8b-9c6c-012cc934484b Enrolled 2026-06-04
SAGE-SQL 120ba7bf-8544-48a0-98a1-40ed5cdd3e1f Enrolled 2026-06-04
DF-HYPERV-B (see RMM dashboard) Enrolled 2026-06-04
DF-SVR-D2-Sync (see RMM dashboard) Enrolled 2026-06-04
ENG-DEV-SERVER (see RMM dashboard) Enrolled 2026-06-04
DFORTH-Ship db17e069-2948-4cbc-97ea-1da721edcaf5 Shipping-station PC; recurring TDR BSOD — see Patterns
(36 additional agents) — Mix of workstations; full list in GuruRMM dashboard

Cannot be enrolled:

Host Reason
MYDATA TPSys (myserver) Fedora Core 3: glibc ~2.3.5 (<2.17 floor), kernel 2.6.16 (<2.6.32 floor), SysV init (no systemd unit target) — three independent hard blockers. Confirmed 2026-07-04.

Backup Architecture

  • MSP360 ("ACG-Online Backup", cbb.exe): Backup provider. Storage account: ACG-Dataforth (account ID 0b49ca5e-...).
  • AD2: Two plans — AD2 Image (image plan, bunch 35a5c3d2, running daily), Files plan (180-day retention, NBF, daily 2 AM, covers C:\Shares tree; GFS off, synthetic full, compression, fast-NTFS). No shadow copies on AD2.
  • AD1: Image2025 image plan + Files plan created 2026-06-05 (NBF, daily 2 AM, 180-day retention, ACG-Dataforth, covers C:\Engineering + C:\Shares\ITSvc; initial run at 2:00 AM, not manually triggered). Both image and file plans now in place, matching AD2.
  • Pre-attack backup (offline, not MSP360): HGHAUBNER D: drive holds a full pre-attack snapshot of all 7 mapped DF shares, captured before the 2025 ransomware event. This is the only recovery source predating the attack. Accessible via GuruRMM user_session on HGHAUBNER. Cross-machine writes use existing GPO-mapped drives only (fresh UNC blocked by WTS-impersonation — see Patterns).
  • Historical file-level backup: NBF bunch faad5a67 ("Backup plan on 8/29/2025") in ACG-Dataforth storage contains restore points 8/29–9/29/2025, archived at old physical path D:\c-drive\... (pre-migration layout). Used successfully 2026-06-04 to confirm SP1366 file contents.
  • WizTree backup CSV (2026-06-04): Full-drive WizTree export of HGHAUBNER's D: stored at AD2 C:\ClaudeTools\clients\dataforth\WizTree_20260604184904.zip (sensitive — kept OFF shares). ~8.7M files / 5.7 TB across 7 shares documented.

Key Applications

Application Host URL/Port Notes
TestDataDB AD2 http://192.168.0.6:3000 Node.js + Express, PostgreSQL 18.3. Internal LAN only. Redesigned UI deployed 2026-06-18 (cert-fit, publish chips, push toasts, full-screen results). Row counts per the 2026-07-01 audit: test_records 475,553, work_orders 34,149, work_order_lines 64,051; 99.3% of test_records have api_uploaded_at set (HTTP API uploader is the live web-delivery path — the older For_Web/ASP.NET path has been dead since 2026-05-11).
Sage ERP SAGE-SQL \SAGE-SQL\sage (S:) RDS-served RemoteApp
GageTrak DF-GAGETRAK (192.168.0.102) — Calibration tracking. Sends email via calibration@dataforth.com (SMTP). GuruRMM enrolled.
Dataforth Product API Hoffman's servers https://www.dataforth.com/api/v1/TestReportDataFiles OAuth2 client_credentials. Vault: clients/dataforth/api-oauth.sops.yaml. Used actively to recover DSCA33/45 and 8B/5B/SCM spec templates.
QuickBASIC 4.5 ATE 64 DOS stations T:\ (\D2TESTNAS\test) Automated test equipment programs. 1,470+ product model specs. Inbound spec/software distribution to stations is currently broken (root-caused 2026-07-01 — see DOS Test Station Data Pipeline pattern and Syncro #32489).
Power Monitor SPA Georg's dev / TBD — Vanilla-JS SPA for Dataforth power meters (built by Georg/Antigravity AI). Demo at PWM.dataforth.com proposed; gateway architecture designed. Parked pending Mike↔Georg conversation. clients/dataforth/power-monitor-demo/
MYDATA TPSys MYDATA TPSys controller (myserver, VLAN 2) local X UI / local PostgreSQL SMT pick-and-place line control software. Newly documented 2026-07-04 — see Infrastructure Servers table.

Syncro Asset Inventory (2026-06-02 Reconciliation)

Pulled full Syncro asset list for customer_id 578095: 78 assets across 2 pages. Syncro currently shows 50 managed assets (confirmed again 2026-07-04 live pull); reconciliation/cleanup ongoing.

Reconciliation Result

Bucket Count Meaning
KEEP 20 Active in Syncro (<150 days since last check-in)
SAVE + FLAG 21 Alive in ScreenConnect or Bitdefender but Syncro agent broken; do NOT delete — reinstall agent
REMOVE 28 Dead in all three systems (Syncro + ScreenConnect + Bitdefender)
VERIFY 9 Servers with no agent anywhere; could be live console-only; confirm before removing

Governing rule (Howard's 3-system OR): A machine is saved if it has been online within 150 days in ANY of Syncro, ScreenConnect, or Bitdefender. Removal only if dead in all three.

SAVE + FLAG — alive but Syncro agent broken (21 machines)

AD1, AD2, SAGE-SQL, FILES-D1, ENG-DEV-SERVER, D2-MFG-001, D1-ENGI-012, MY9-PC, D1-CUST-003, DANC0619, DFORTH-SHIP, DF-LEE11-I9, DFASLB0519, D2-AS-26, HGHAUBNER, D1-PWRM, D1-ENGI-EMCLAB1, D1-CONF-002, D2-HIPOT-SURFAC, D2-AS-34, TS-41 (shows as STATION_41 in ScreenConnect)

VERIFY — servers with no agent (9 machines)

APPS, EXCHANGE, EXCHANGE16, AD-3, AD-4, OLD-AD2, SAGETS-1, EPICOR, D2-ASSY-001

Likely dead: OLD-AD2, EXCHANGE16, SAGETS-1. Confirm before removing: APPS, AD-3, AD-4, EXCHANGE, EPICOR, D2-ASSY-001.

REMOVE — confirmed dead in all systems (28 asset IDs)

Syncro asset IDs: 23845, 149614, 9708445, 9357407, 9276901, 9212922, 9078651, 8824875, 8824867, 8726494, 8726485, 8657233, 8606209, 8572160, 8523941, 8411908, 8410614, 8632009, 8726495, 8421223, 9081717, 8726493, 8423782, 8726481, 8525650, 8622969, 8361459, 8670944

Deletion method: Syncro GUI only (https://computerguru.syncromsp.com/customer_assets?customer_id=578095). API route DELETE /customer_assets/{id} returns HTML 404 for this integration token — not exposed.

Root Cause — Fleet-wide Syncro Agent Break ~2025-10-06

57 of 78 assets show updated_at frozen at or before 2025-10-06, while the remaining 21 show recent check-ins. This is a hard cutoff, not gradual attrition — indicating a fleet-wide Syncro agent failure around that date. The machines stayed online (visible in ScreenConnect); only the Syncro agent stopped reporting. Root cause not yet investigated.

Pending Actions (Coord todo tree, parent 103c48ad-7b31-4967-9388-065a91888e7c, assigned to Howard)

  1. Delete the 28 confirmed-dead assets in Syncro GUI.
  2. Decide the 9 VERIFY servers.
  3. Reinstall Syncro agent on the 21 SAVE+FLAG machines.
  4. Switch Dataforth to metered Syncro asset billing once clean.
  5. Reply to Winter; flag the ~2025-10-06 fleet-wide agent break for investigation.

Third-Party Tool Inventory

Bitdefender GravityZone

  • Company ID: 64c94ef310db128bfa0d908f (suffix _578095 confirms Dataforth mapping)
  • Status: Dataforth is being phased off Bitdefender. Only 4 of 57 GravityZone endpoints remain in "Custom Groups" (actively managed); 53 are in the "Deleted" folder (mostly unmanaged).
  • [WARNING] Bitdefender absence is NOT a decommission signal for Dataforth. A machine missing from BD may simply have had its BD agent uninstalled as part of the phase-off. Use Syncro or ScreenConnect as liveness indicators.
  • GravityZone company owner field: Lee Payne.

ScreenConnect

  • Host: https://computerguru.screenconnect.com
  • Extension GUID: 2d558935-686a-4bd0-9991-07539f5fe749
  • Vault: msp-tools/screenconnect.sops.yaml (fields credentials.username, credentials.api_secret)
  • Working API auth (determined 2026-06-02): CTRLAuthHeader: <raw api_secret> (NO "Basic " prefix) + Origin: https://computerguru.screenconnect.com. Basic-auth or "Basic " in CTRLAuthHeader both return 401.
  • Only exposed method: POST /App_Extensions/<guid>/Service.ashx/GetSessionsByName with body {"sessionName":"<name>"}. All other Get* method names return 500. Agent Name fields are blank for unattended sessions — this API cannot enumerate the full Dataforth fleet; name-based lookup only.
  • Custom session properties: CP1=Company, CP2=Site, CP3=Tag.

Access

Domain / Server Access

  • AD2 SSH: ssh sysadmin@192.168.0.6 (port 22) — vault: clients/dataforth/ad2.sops.yaml → credentials.password — NOTE: stale backslash escape in vault entry; strip with sed 's/\\//g'. MTU-sensitive: GURU-5070 OpenVPN adapter ifIndex 12 must be MTU 1400 for reliable bulk transfers.
  • AD1 SSH: ssh sysadmin@192.168.0.27 — vault: clients/dataforth/ad1.sops.yaml
  • D2TESTNAS SSH: ssh root@192.168.0.9 — vault: clients/dataforth/d2testnas.sops.yaml. Use root, NOT sysadmin (sysadmin SSH fails on D2TESTNAS). SSH key from acg-guru-5070 authorized. From AD2, root SSH key auth to D2TESTNAS is currently broken (publickey denied, confirmed 2026-07-01) — the live rsync-daemon sync path does not depend on it, but fix or retire the dormant SCP script that does.
  • D2TESTNAS aoibackup share (AOI XP backup): \\192.168.0.9\aoibackup — Samba user admin (password matches the XP's local login), hosts allow = 192.168.1.175 only, browseable = no. Other NAS shares explicitly deny 192.168.1.175. Creds in vault: clients/dataforth/d2testnas.sops.yaml → credentials.smb.aoi-user / .aoi-password / .aoi-share.
  • UDM SSH: ssh azcomputerguru@192.168.0.254 (2FA push) or ssh root@192.168.0.254 (root SSH key installed 2026-06-08). Jump via D2TESTNAS: paramiko direct-tcpip channel or ProxyJump. Vault: clients/dataforth/udm.sops.yaml (corrected 2026-06-09).
  • SAGE-SQL SSH: ssh sysadmin@192.168.0.153 — SSH key (C:\ProgramData\ssh\administrators_authorized_keys on SAGE-SQL)
  • All server passwords: vault (individual vault entries per server — clients/dataforth/<host>.sops.yaml)
  • WinRM (AD2/AD1): port 5985 — pywinrm with NTLM, user INTRANET\sysadmin
  • HGHAUBNER: No SSH. Reached via GuruRMM agent 2aefe0d5. Logged-in user intranet\ghaubner. Cross-machine file writes use existing GPO-mapped drives only (Q: → \ad2\c-drive, T: → \ad2\e-drive, etc.).

MYDATA TPSys SMT Controller (new, 2026-07-04)

  • Root: password — vault clients/dataforth/mydata-smt.sops.yaml. Console/network access method beyond physical console not yet documented (verify — likely SSH once IP is confirmed). Exact IP on VLAN 2 "mydata" (192.168.1.0/24) not yet confirmed (verify).
  • Recovery method if locked out again: at the LILO boot: prompt, boot linux init=/bin/bash rw to land in a passwordless root shell (bypasses sulogin, which on this Red-Hat-family box would otherwise demand the root password even in single-user mode), then mount -o remount,rw / and passwd root. Reboot with reboot -f or, if that hangs, echo b > /proc/sysrq-trigger.
  • Accounts: root, tpsys (TPSys application user, being added to wheel + scoped NOPASSWD sudo for the app-launch command), tpspool (TPSys spool user), postgres (uid 500, local TPSys database).
  • No prior credential existed for this machine in vault or wiki before 2026-07-04 — it was undocumented until discovered at the physical console.

M365 / Entra

  • M365 admin: sysadmin@dataforth.com — vault: clients/dataforth/m365.sops.yaml
  • Tenant ID: 7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584
  • Claude-Code-M365 Entra App: App ID 7a8c0b2e-57fb-4d79-9b5a-4b88d21b1f29, secret expires 2027-12-22 — vault: clients/dataforth/m365.sops.yaml → credentials.entra-app
  • MSP remediation app suite: MSP tenant ce61461e-81a0-4c84-bb4a-7b354a9a356d — tiered ComputerGuru apps (Exchange Operator b43e7342 etc.), vault msp-tools/computerguru-*.sops.yaml. (Old single app fabb3421/Claude-MSP-Access DELETED 2026-06-14 — AADSTS700016, do not use.)
  • ComputerGuru tiered apps: All 5 apps consented 2026-04-23. Exchange Operator SP (b43e7342) had Exchange Admin role added manually (gap in onboard-tenant.sh — not auto-assigned for Exch Operator).

MSP360 Managed Backup API

  • Vault: msp-tools/msp360-api.sops.yaml (api.mspbackups.com, /api/Provider/Login)
  • cbb.exe path on AD2: C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe
  • Browse file backup: cbb.exe list -a "ACG-Dataforth" -b <bunch_id> -rp <restore_point_id> -path "<path>"

Dataforth Product API (Hoffman)

  • Vault: clients/dataforth/api-oauth.sops.yaml
  • Token URL: https://login.dataforth.com/connect/token
  • Grant: client_credentials, Client ID: dataforth.onprem.sync, Scope: dataforth.web
  • Token TTL: 1 hour
  • Swagger: https://www.dataforth.com/swagger/index.html
  • Endpoints: GET /api/v1/TestReportDataFiles/{serial} (per-model cert), /bulk, /stats

ESXi / Hypervisors

  • ESXi-122: 192.168.0.122 — vault: clients/dataforth/esxi-122.sops.yaml
  • ESXi-124: 192.168.0.124 — vault: clients/dataforth/esxi-124.sops.yaml

PBX

  • Vault: clients/dataforth/pbx.sops.yaml (password corrected 2026-06-23 — see Infrastructure Servers table)
  • SSH: sangoma@192.168.100.2

Patterns & Known Issues

Active Directory

  • No custom security groups — only default Windows groups. Service accounts in OU=ServiceAccounts.
  • ClaudeTools-ReadOnly AD account — purpose unclear. Investigate.
  • Ken Hoffman has two accounts (khoffman + oemdata) — not consolidated.
  • jlohr account retained — post-retirement (2026-03-31), kept enabled specifically to receive ntirety.com infrastructure notifications. Inbox rule forwards to mike@azcomputerguru.com. Do NOT disable.
  • Entra sync scope: OU=SyncedUsers and OU=Azure_Users sync to Entra (cbell confirmed in OU=Azure_Users, synced — 2026-06-01; the prior "SyncedUsers only" note was incomplete). CompanyUsers OU does NOT sync. 38 stale TS-* test station accounts were cleaned from Entra 2026-03-27.

RDS / SAGE-SQL

  • RDS licensing: Grace period reset 2026-05-06 by deleting GracePeriod registry key. Grace period expires again without proper CALs. Purchase RDS CALs (Per User mode, LicensingType=4).
  • TSGateway: Disabled on SAGE-SQL (server not externally exposed at firewall). Do NOT re-enable without reason.
  • SSL cert: Self-signed, subject CN=sage-sql.intranet.dataforth.com. Non-domain machines must manually import to Trusted Root + Trusted Publishers.
  • GPO cert distribution: Not completed (AD2 SYSVOL write blocked from non-domain workstation). Pending.
  • Bitdefender GravityZone: Managed AV on SAGE-SQL. Can block PowerShell execution — may need temporary disable for admin work.

Voice / Phones / FreePBX

  • Production phones VLAN: 192.168.100.0/24. PBX at .196 / .2. All production phones live here.
  • Unifi default voice VLAN (192.168.1.0/24): NOT used for production voice — phones landing here cannot reach PBX. Switch port misconfiguration symptom: phone shows wrong date/time (NTP failure) and no dial tone. (This subnet is however in active use for the mydata/SMT VLAN — see Network section; do not confuse the two purposes.)
  • D1-Server-Room port 1: Controls lobby drop → must stay on VLAN 100. Reverted to default once before (2026-05-04 incident).
  • FirstDigital trunk — qualify_frequency=0: FD's Sonus SBC ignores SIP OPTIONS keepalives. Setting qualify=0 in the pjsip DB (id=1) prevents trunk from going Unavailable. Do NOT revert to a non-zero qualify. (Total phone outage 2026-06-08 was caused by FD SBC not answering OPTIONS, making trunk go Unavailable and blocking all INVITEs.)
  • PJSip.class.php line 504 patch must be re-applied after any fwconsole ma updateall. It is wiped by FreePBX updates. Backup before each update (PJSip.class.php.bak.<timestamp>).
  • Do NOT port-forward the RTP range (10000-20000) on the UDM for this trunk. A static RTP DNAT creates a conntrack collision with the PBX's outbound RTP — inbound works but outbound audio dies. SIP 5060 forward only (source-locked to 66.7.123.0/24). Current on_boot.d script (30-freepbx-sip-forward.sh) is SIP-only, correct.
  • Inbound SIP relies on /data/on_boot.d/30-freepbx-sip-forward.sh — not a persistent UniFi UI rule. Must survive UDM reboot via the script. Confirmed 2026-06-23: the risk window is broader than "survives reboot" — a mid-uptime UniFi controller provision/update can also flush the SIP DNAT while the box never reboots, silently killing inbound calls until the script is manually re-run (sh /data/on_boot.d/30-freepbx-sip-forward.sh). Diagnostic: iptables -t nat -S | grep -iE '5060|192.168.100.2' on the UDM — empty output means the rule is gone. Recommend Mike add a UI port-forward as a belt-and-suspenders measure (not yet done as of 2026-07-04).
  • PBX sudo authorization gap: sangoma is in the sudo group but sudo authorization on this distro appears not actually granted (a sudo-based liveness test can fail on authorization, not auth — do not use it to judge whether a password/credential is stale).

Exchange Online / Email

  • INKY PhishFence StopProcessingRules: Kills all subsequent transport rules. Use inbox rules for per-mailbox forwarding, NOT transport rules.
  • Mailprotector CloudFilter: Outbound delivery goes through Mailprotector. If a message is "Delivered" per Dataforth's outbound trace but never arrives, check Mailprotector (/mailprotector skill, py mp.py messages ...) — it may be held. The INKY "Annotation - Recipient Not Group Member" transport rule can route mail to Mailprotector's hold queue.
  • AutoForwarding blocked by default (tenant outbound spam policy). If per-user forwarding needed, create scoped HostedOutboundSpamFilterPolicy for that sender with AutoForwardingMode=On.
  • Get-MessageTrace deprecated Sept 2025: Use Get-MessageTraceV2 and Get-MessageTraceDetailV2 in Exchange PowerShell.

GuruRMM Agent Deployment

  • WebSocket auth bug (Issue #8): enrolled_agents.agent_key_hash is never checked by ws/mod.rs. Workaround: after MSI install, overwrite registry HKLM:\SOFTWARE\GuruRMM\AgentKey with the site API key (not enrollment AgentKey), then restart service.
  • rmm-api.azcomputerguru.com must be grey-clouded (DNS-only, not proxied) — Cloudflare proxy blocks WebSocket. Do NOT re-enable orange cloud. Gitea Issue #9.
  • No RMM agent possible on FC3-class legacy appliances (confirmed 2026-07-04, MYDATA TPSys): the Linux agent installer requires glibc ~2.17+, kernel >=2.6.32, and installs a systemd unit. A box on glibc ~2.3.5 / kernel 2.6.16 / SysV init (Fedora Core 3 "Heidelberg", Nov 2004) fails all three floors simultaneously — do not attempt install; plan agentless monitoring (ICMP/TCP probe or SSH heartbeat from a host that can reach the appliance's VLAN, e.g. D2TESTNAS or the RMM server, since inter-VLAN routing from mydata to main LAN is open) instead. If this becomes a recurring need across other legacy appliances, file a /feature-request for legacy/appliance Linux monitoring support in GuruRMM.
  • Legacy Red-Hat-family recovery pattern: if a box like this is ever locked out again with no vaulted credential, LILO boot: → linux init=/bin/bash rw gives a passwordless root shell and bypasses sulogin (which on this OS family would otherwise demand the root password even in single-user mode). Always check BOTH vault and wiki before assuming a machine has no documented credential — a prior session missed the wiki check here and had to be corrected.

Cross-Machine File Operations (Windows Domain)

  • Double-hop / WTS-impersonation blocks fresh UNC paths. When running commands in GuruRMM user_session (or via SSH-through-another-server), the impersonated token carries no network credentials. net use and fresh \\server\share paths fail with Access Denied.
  • Workaround that works: Run on the SOURCE machine in user_session and write to an existing GPO-mapped drive (e.g. Q: → \\ad2\c-drive). The existing mapping survives impersonation; fresh UNC does not.
  • Proven 2026-06-04 on HGHAUBNER: local D:\DF C-Drive read + Q: write succeeded; AD2-side user_session copy and SSH-from-AD2 both failed.

AD2 SSH / VPN MTU

  • PMTU blackhole on GURU-5070 → AD2 SSH: GURU-5070's OpenVPN adapter "Local Area Connection" (ifIndex 12, IP 192.168.6.2) defaults to MTU 1500. Tunnel path MTU is ~1424 (FD ping confirms). Over-MTU bulk TCP segments (SSH transfers, SCP) are silently dropped. Small interactive commands pass, creating a false appearance of "flaky VPN" or "SSH ban."
  • Fix (applied 2026-06-18): Set-NetIPInterface -InterfaceIndex 12 -AddressFamily IPv4 -NlMtuBytes 1400 on GURU-5070 via SYSTEM RMM agent. Registry-persistent but may reset on OpenVPN reconnect — verify with Get-NetIPInterface -InterfaceIndex 12.
  • Durable fix: server-side mssfix 1360 on the Dataforth OpenVPN server (or push "tun-mtu 1400") — would auto-clamp all fleet clients, not just GURU-5070.
  • AD2 is NOT the target for SSH diagnosis when SSH is the failing channel — use RMM instead.

AD2 Branch / Coordination

  • AD2 operates on the ad2 git branch. Fork is rebased from main + thin Dataforth-specific commits. Do NOT edit shared fleet files on ad2 — conflicts on every sync. Dataforth context lives in clients/dataforth/CLAUDE.dataforth.md.
  • AD2 is coord-API isolated: 172.16.3.30 is unreachable from Dataforth LAN. Coord messages, locks, and todos NEVER reach AD2. All inter-session coordination goes through git sync: committed handoff docs + ## Note for <user> blocks. Do NOT use the coord skill for AD2.
  • sync.sh on AD2: not fork-aware on the push step (always tries main); force-push manually: git push --force-with-lease origin ad2 after rebasing.
  • New (2026-07-01): AD2's GuruRMM agent can be used as a live read-only ground-truth channel by spawning a headless claude -p through it (context:user_session, detached, poll for a DONE marker) — bypasses the coord isolation for READ-ONLY investigation without waiting on git-sync round trips. Unset the stale machine-level ANTHROPIC_API_KEY env var first or auth fails. See RMM-Spawned Claude above.

Post-Ransomware Recovery Restore (2025) — Incomplete File Migration

  • The 10/1/2025 recovery restore was incomplete. The Restore plan 10/1/2025 (~3.4M files) migrated each share from the old D:\<share> layout to the current C:\Shares\... layout on AD2 and dropped files in the process. Proven case: SP1366 MAQ20 Communications Module — each PRINTOUTS FOR MANUFACTURING folder for revisions E–H received only one file (the drill panel) when the backup contained ~6 files per revision. The 9/29/2025 file-level backup confirms the files existed before the restore.
  • Scope unknown. Other folders across the 7 shares may have similar gaps. A full migration-gap audit is underway (WizTree both sides — see Active Work). The audit is review-only — no automatic restore, because some deletions were intentional and the HGHAUBNER backup is additive-only (includes Georg's personal files alongside corporate data).
  • Backup-side CSV for diffing stored at AD2 C:\ClaudeTools\clients\dataforth\WizTree_20260604184904.zip (sensitive file list — keep off shares and off any publicly accessible directory).
  • AD2 D: drive is gone. The old D:\c-drive data volume was repurposed as a mounted Windows install ISO during the rebuild. All share data now lives under C:\Shares. The historical file-level backup (bunch faad5a67) archived the data under D:\c-drive\... (pre-migration path) — reconcile paths accordingly.

Shares ACL State — All Open to All Staff

  • All 8 business shares grant access to every employee via Everyone/Domain Users (FullControl on 4 shares, Modify on 3). No department-based security groups exist. Sensitive data — Payroll, OSHA records, Purchase Orders, Accounting/QuickBooks, Sage financials — is fully readable and writable by all domain users.
  • Remediation project in progress (Shares & Permissions, started 2026-06-10). Phase 0 (discovery) complete. Phase 1 (client input/department matrix) pending email to Dan Center. Do not apply ACL changes until after client sign-off on the target model. Details: clients/dataforth/docs/projects/shares-permissions/.
  • Special shares excluded from remediation: test (DOS/SMB1 guest — leave open); webshare (preserve svc_testdatadb:Full); ITSvc (Domain Computers needs Read); Sage app data path (restrict by group at the share, but keep the live UNC stable for the ERP/SQL).
  • Phase 2 target-state strawman (drafted 2026-06-22, pre-client-input): target-structure-draft-2026-06-22.md. Inferred from the existing share/folder layout (which is already department-shaped) plus a client-facing render at Dataforth-Shared-Drives-Plan.html. Target = one logical tree: Company\Departments\ (Engineering [+Test-Engineering], Manufacturing, Quality, Sales-Marketing, Shipping-Receiving, Purchasing, IT), a Restricted\ branch with broken inheritance / no Domain Users (Accounting-Finance, Payroll, HR, OSHA, Purchase-Orders), a read-mostly Company-Wide\, per-user Users\, and read-only Archive\. ABE on. Groups named SG-<Resource>-<RW|RO>; users get Modify via the RW group (never Full), SYSTEM/Administrators keep Full.
  • Drive-letter strategy — Option A recommended: keep current Q/S/T/W/Y/B mappings and realize the tree logically (reorg folders within each share + apply groups) for the first rollout — lowest disruption, no app/UNC breakage, no retraining. Hold physical consolidation to one Company drive (Option B) as a later optional phase after a hard-coded-UNC-path audit (DOS, Sage, datasheet pipeline, GageTrak/Epicor). The permission model is identical either way.
  • Strawman is NOT a build order — six items still gate Phase 2 sign-off (need the client): confirm the inferred department list; the per-department RW/RO/none access matrix; named access for sensitive data (Payroll/OSHA/POs/Accounting — likely HR/Finance sign-off, not just Dan); department rosters to populate groups; legacy cleanup approval (person-named / "Do not use" folders); and an Engineering destination volume (AD1 C: ~90% full blocks any ENGR restructure).

DOS Test Station Data Pipeline (new, 2026-07-01 ground-truth audit)

  • Root cause of Syncro #32489 confirmed (F1, HIGH): the deployed inbound spec downloader T:\COMMON\ProdSW\NWTOC.BAT v5.0 (mirrored to NAS COMMON/ProdSW/NWTOC.BAT) copies only *.BAT and *.EXE from the NAS to stations — zero .DAT files. Its own changelog header says so verbatim: "Added EXE copy, removed DATA folder copies (avoid cyclic overwrites)". No version of NWTOC, past or present, has ever distributed the shared COMMON master specs — the fix must ADD a data copy, not "restore" one. Engineering masters (e.g. 5BMAIN.DAT, updated 2026-06-26) reach the NAS fine; they simply never reach the stations.
  • New risk found (F2, HIGH, needs on-station confirmation): the deployed NWTOC.BAT v5.0 and CTONWTXT.BAT v2.3 use COPY /Y. /Y is not a valid MS-DOS 6.22 switch (introduced in MS-DOS 7.0/Windows 95); 6.22's COPY supports only /A /B /V. If the stations run genuine 6.22, COPY /Y returns Invalid switch - /Y and copies nothing — meaning NWTOC has been silently copying NOTHING (not even the .BAT/.EXE files it's supposed to) since it was deployed 2026-03-16. This was independently confirmed by Grok in verify mode. The pivotal unresolved question — whether the stations run true 6.22 or MS-DOS 7.x — is empirical and can only be checked on a station itself (VER, then COPY /Y NUL C:\TEST.TXT); stations have no RMM agent. The upload path (CTONW.BAT v5.0) is unaffected — it uses plain COPY — which is why test data has kept flowing even if NWTOC is dead.
  • Do NOT judge DOS-6.22 compatibility from the root-level test\{NWTOC,CTONW,CHECKUPD,STAGE}.BAT v1.x files — those are abandoned drafts riddled with NT-only constructs (FOR /F, SET /A, CALL :label, 2>NUL stderr redirection, tilde path modifiers). The scripts that actually run on stations live under COMMON\ProdSW\ and were deliberately cleaned of those constructs (except for the /Y issue above).
  • F3 (MED): C:\Shares\test\TS-21\ProdSW is a stray file (a misplaced 7BMAIN4-type EXE), not the directory rsync expects — this makes the 15-minute AD2↔NAS sync report ERRORS on every single run (exit 3, "Not a directory"), masking any genuine new failure. Fix: remove/relocate the file, harden the push loop to skip non-directory ProdSW paths.
  • F4 (MED): Server-side datasheet generation (spec-reader.js) reads specs from testdatadb\specdata\, which is a frozen 2026-03-27 snapshot — not the live engineering masters. Any spec limit changed after that date is not reflected in generated datasheets even though the outbound data pipeline itself is healthy.
  • F5 (MED, security): Plaintext credentials found hard-coded in scripts on AD2 — rsync daemon password in Sync-FromNAS-rsync.ps1, NAS root SSH password in the dormant Sync-FromNAS.ps1, and the Postgres testdatadb_app password in testdatadb\database\db.js. Flagged for rotation + proper vaulting; not yet remediated as of 2026-07-04.
  • Stale-assumption corrections established by this audit (do not carry forward the old versions): datastore is PostgreSQL 18, not SQLite (the SQLite file is a 4.4 GB archive from the 2026-04-03 migration cutover); the scheduled sync task runs Sync-FromNAS-rsync.ps1, not the dormant per-file SCP script; web delivery is a live HTTP API uploader (upload-to-api.js, 472,290 records flagged as of 2026-07-01), not the dead For_Web/ASP.NET path (dead since 2026-05-11); CTONWTXT.BAT IS actively invoked (called from CTONW.BAT line 30), contradicting an earlier assumption that it was a gap.
  • Full report: clients/dataforth/docs/audits/2026-07-01-test-data-chain-audit-AD2.md. Recommendations in that report are proposals only — nothing has been applied (the audit was strictly read-only).

Hardware / Endpoint — Aging Fleet

  • DFORTH-Ship recurring TDR BSOD (0x116 VIDEO_TDR_FAILURE), diagnosed 2026-06-25: integrated Intel HD Graphics 4600 on driver 20.19.15.5126 (Intel's final driver for that part, dated 2020-01-20) hitting the GPU-reset timeout on an 11.5-year-old HP EliteDesk 800 G1 USDT (BIOS 2014-12-10). Five minidumps span 2025-11-03 through 2026-06-24 with an accelerating cadence — treat as a degrading-hardware trend, not a one-off. Mitigation applied: Edge hardware acceleration disabled via machine policy (HKLM\SOFTWARE\Policies\Microsoft\Edge\HardwareAccelerationModeEnabled = 0). No durable fix is possible for integrated graphics (nothing to reseat/replace) — PC replacement is the real fix; thermal cleaning of the USDT chassis is a secondary mitigation worth doing regardless. Do not confuse with near-twin host DForth-Shipp — verify the exact agent ID before acting.

Security

  • C2 IP blocks are iptables only — do not survive UDM reboot. Must add to permanent UniFi block list via UI. C2 IPs: 80.76.49.18, 45.88.91.99 (AS399486 Virtuo, Montreal).
  • AD1 disk 90% full — C:\Engineering = 787 GB of 1023 GB. Risk of replication failures.
  • Windows Firewall disabled on AD2 (all profiles) — known risk, not yet remediated.
  • 3 Windows 7 machines on network (LABELPC, LABELPC2, D2-RCVG-003) — EOL, unpatched.
  • AD1/AD2 on Windows Server 2016 / 2019 respectively — approaching/at end of mainstream support. Plan upgrade.
  • Entra ID P2 not licensed — IdentityRiskyUser risk check returns 403 even with scope consented. Would need P2 upgrade to enable Identity Protection.
  • IdentityRiskyUser.Read.All scope: Consented to Security Investigator app but unusable (no P2 license).
  • Plaintext credentials in Dataforth test-data-chain scripts (F5, 2026-07-01): rsync daemon, NAS root, and Postgres app passwords are hard-coded in scripts under C:\Shares\test\scripts\ and testdatadb\database\db.js, world-readable via the test SMB share. Rotation + vaulting recommended, not yet done.
  • Legacy appliance with no prior credential (MYDATA TPSys, 2026-07-04): an entire production SMT-line controller existed with no vault or wiki entry until physically discovered. Root password reset via LILO recovery; now vaulted. Worth a broader sweep for other undocumented devices on VLAN 2 "mydata" given 3 unnamed industrial MACs were already known but unresolved as of 2026-06-01.

Syncro Asset Management

  • Fleet-wide Syncro agent break ~2025-10-06: ~half of Dataforth machines stopped reporting to Syncro on or around that date while remaining online in ScreenConnect. Do NOT auto-remove machines frozen at that date without cross-checking ScreenConnect. Root cause unknown — needs investigation.
  • Bitdefender is NOT a liveness signal: Dataforth is being phased off BD; 53 of 57 GravityZone endpoints are in the "Deleted" folder. Missing from BD = BD agent uninstalled, not machine dead.
  • API delete not available: DELETE /customer_assets/{id} returns HTML 404 for the current integration token. All asset deletions must go through the Syncro GUI.

staff Share Missing

  • The staff network share is absent from FILES-D1 (only archive and sales exist). HGHAUBNER's backup includes a DF Staff folder, suggesting the share existed pre-attack. Not in scope for the current migration-gap diff — separate issue requiring investigation.

Active Work

As of 2026-07-04 (0 open Syncro tickets per live pull):

  • DOS Test Station Data Pipeline (Syncro #32489, active): Root cause confirmed 2026-07-01 via a read-only ground-truth audit run through a headless Claude spawned on AD2's GuruRMM agent (new capability — see RMM-Spawned Claude). F1 (NWTOC v5.0 never copies master .DAT specs to stations) is confirmed; F2 (COPY /Y may not be valid on true MS-DOS 6.22) needs a station-side check before scoping the fix. Next steps: (1) confirm station DOS version (VER + COPY /Y NUL C:\TEST.TXT on a station), (2) draft a DOS-6.22-safe NWTOC v5.1 that adds a one-way pull of master .DATs (plain COPY, no /Y if 6.22 confirmed) without reintroducing the cyclic-overwrite problem v5.0 was avoiding, (3) Grok-review the new script before it touches a station, (4) update ticket #32489 with the confirmed root cause and plan. Secondary cleanup items from the same audit (not urgent): remove the stray TS-21\ProdSW file (F3), feed testdatadb\specdata\ from live engineering masters (F4), rotate/vault the plaintext creds found in scripts (F5), retire dead For_Web output and abandoned v1.x script drafts (F6/F7/F8).

  • MYDATA TPSys SMT controller (new, discovered 2026-07-04): Root password reset via LILO recovery and vaulted 2026-07-04 at clients/dataforth/mydata-smt.sops.yaml (host/VLAN/OS/accounts/recovery-method documented; decrypt-verified). Outstanding: (1) confirm the machine's exact IP on 192.168.1.x and reconcile against the known mydata VLAN member list; (2) verify the tpsys wheel-group + scoped NOPASSWD sudoers change actually landed (id tpsys, sudo -l as tpsys); (3) get the exact TPSys app-launch command from Howard/Mike to finalize the sudoers scope; (4) confirm the controller booted cleanly into TPSys after the forced reboot (it is a live production SMT line); (5) decide and stand up agentless monitoring (ICMP/TCP probe or SSH heartbeat from D2TESTNAS or the RMM server — inter-VLAN routing to mydata is open) since a GuruRMM agent is impossible on this OS; formalize via /feature-request if Mike wants legacy/appliance Linux monitoring as a standing GuruRMM capability.

  • DFORTH-Ship BSOD (ongoing monitoring): Edge hardware-acceleration mitigation applied 2026-06-25; needs on-site Edge restart/reboot to take effect, verify at edge://policy. Monitor for recurrence — if it bugchecks again, pull and analyze the four older dump signatures to confirm whether it is drifting toward a hard hardware fault. Schedule thermal cleaning of the USDT chassis. Recommend/plan replacement of the 11.5-year-old EliteDesk 800 G1 USDT shipping station as the durable fix.

  • UDM inbound SIP DNAT (recurring risk, unresolved): Confirmed again 2026-06-23 that the SIP 5060 DNAT can be flushed by a UniFi controller provision mid-uptime, not only at reboot. Coord to-do 45572ee1 tracks the durable fix (persistent UI port-forward rule or a cron/watcher re-running the idempotent on_boot.d script) — needs a maintenance window. Still SIP-only; never forward the RTP range.

  • Shares & Permissions project (Phase 1 — BLOCKING, pending client input): Phase 0 (discovery) completed 2026-06-10 — read-only ACL audit confirmed all 8 business shares open to all employees; Domain Users has FullControl on 4 shares. Discovery email to Dan Center drafted (clients/dataforth/docs/projects/shares-permissions/discovery-email-draft.md); not yet sent — recipients/sender not locked (Dan Center primary; CC Kevin Wackerly?; Mike or Howard sending?). Phase 1 blocked on client responses: department list, access matrix, sensitive-data rules, staff rosters. A Phase 2 target-state strawman was drafted 2026-06-22 (target-structure-draft-2026-06-22.md + client-facing Dataforth-Shared-Drives-Plan.html) from the existing layout — see Shares ACL State; it still needs the Phase 1 client matrix to finalize. Next-step options: polish the client HTML, finalize + send the discovery email to unblock Phase 1, or refine the internal strawman. Full roadmap: clients/dataforth/docs/projects/shares-permissions/roadmap.md.

  • 8B/5B/SCM render completion (parked with AD2): Root-caused a parseRawData bug (PASS/FAIL line consumed as step-response for families that omit "0","0",v line). 136 8B/5B/SCM templates mined from Hoffman API (2026-06-18). Completion — wiring templates into the live renderer with correct slotmaps, QB rounding, and frequency/AAC accuracy — handed to AD2 (its now-proven machinery from DSCA33/45 work). Sync handoff at projects/dataforth-dos/8B5BSCM-RENDER-VERIFY-2026-06-18.md. ~9,624 records remain unpublished; this is a render-coverage gap (null renders correctly skipped), not a backlog.

  • Migration-gap audit (parked): WizTree CSV of HGHAUBNER's pre-attack backup captured (AD2 C:\ClaudeTools\clients\dataforth\WizTree_20260604184904.zip). WizTree runs on live servers deferred — no diff yet. Plan: run WizTree on AD2, FILES-D1, SAGE-SQL, AD1 → diff CSV-to-CSV per share → clients/dataforth/migration-gap-catalog-2026-06-04.md. Full plan in clients/dataforth/migration-gap-diff-RESUME.md. No auto-restore — review-only catalog.

  • Syncro asset cleanup (with Howard): 78-asset reconciliation complete. 28 confirmed-dead assets pending GUI deletion; 21 alive-but-broken machines need Syncro agent reinstall; 9 servers in VERIFY bucket. Move to metered billing once clean. Coord todo tree assigned to Howard (parent 103c48ad-7b31-4967-9388-065a91888e7c). See Syncro Asset Inventory above.

  • AOI XP backup + isolation (ongoing): AOI optical-inspection XP PC on VLAN 2 (mydata/SMT) @ 192.168.1.175; locked-down SMB1 share aoibackup on D2TESTNAS (XP-only, user admin). Other NAS shares now deny the XP. Optional EOL hardening pending: block XP → company LAN (except NAS 192.168.0.9) + Internet on the UDM, scoped to .175. Todo 37543f7f.

  • AD2 Claude capability updates (parked): AD2 runs its own Claude from C:\ClaudeTools on the ad2 branch. Needs: (a) syncro + coord commands, (b) DF wiki read-write, (c) Dataforth client data access. Python 3.12.8 and identity.json installed 2026-06-17. Coord API unreachable from Dataforth LAN — comms via git sync only, though the 2026-07-01 RMM-spawn pattern now offers a read-only side channel for investigation.

  • Power Monitor SPA demo (parked): Georg Haubner developed a vanilla-JS power-meter SPA (AI-built, clients/dataforth/ExternalCodeReview.zip). ACG designed a gateway architecture for a gated demo at PWM.dataforth.com (inbound tunnel, no meter publicly exposed, magic-link auth). Spec at clients/dataforth/power-monitor-demo/GATEWAY-SPEC.md. Parked pending Mike↔Georg conversation.

  • Test Datasheet Pipeline:

    • Production pipeline healthy — outbound (station → NAS → AD2 → Postgres → web) confirmed current as of 2026-07-01 (import as recent as 13:41 UTC same day). 475K+ records, DSCA33/45 recovery complete (1,452 new certs published 2026-06-18 via Hoffman API).
    • Inbound spec/software distribution to stations is broken — see DOS Test Station Data Pipeline pattern above and Syncro #32489.
    • Email notifications deployed (Graph API via sysadmin@dataforth.com).
    • 8B/5B/SCM render gap — parked with AD2 (see above).
    • 2 niche DSCA models (DSCA33-1948, DSCA45-1746) and their 8B equivalents have no Hoffman original — no template, cannot auto-publish.
    • DKIM: cutover to selector2 on 2026-05-16 — no action needed; verify signing after that date.

  • GAGEtrak email (ticket #32142): calibration@ SMTP re-enabled 2026-04-23. GAGEtrak configured (smtp.office365.com:587, calibration@dataforth.com). Kevin Wackerly verifying schedule — expected Monday run appears to run Tuesday.

  • jlohr forwarding: ntirety.com inbox rule active as of 2026-05-12; confirmed delivering to mike@azcomputerguru.com. Defunct transport rule pending cleanup.

  • RDS / SAGE-SQL: RDS grace period reset. GPO cert distribution pending. RDS CALs purchase needed long-term.

  • MFA enforcement ongoing — 19 users were not enrolled as of April 4 enforcement date; current enrollment count unverified.

  • C2 IP blocks need permanence: Iptables rules on UDM (80.76.49.18, 45.88.91.99) need to be added to permanent UniFi UI block list.

History Highlights

Date Event
2025 Crypto/ransomware attack — AD2 wiped and rebuilt, many files lost. Test datasheet pipeline broken.
2025-08-29 – 2025-09-29 MSP360 file-level backup (faad5a67) covering DF shares at old D:\c-drive\... path. Last snapshot before the recovery restore.
2025-10-01 – 2025-10-02 Post-ransomware recovery restore (Restore plan 10/1/2025, ~3.4M files) migrated shares from D:\<share> to C:\Shares\... on AD2. Restore was incomplete — files dropped in multiple folders (root cause: restore tool gap, not user deletion). AD2 C:\Shares tree NTFS creation timestamp confirms this date.
~2025-10-06 Fleet-wide Syncro agent break — ~half of Dataforth machines freeze in Syncro while remaining online in ScreenConnect. Root cause unknown.
2026-01-19 DOS Update System built and deployed — NWTOC/CTONW/UPDATE/DEPLOY BAT files, 39 deployments. Sync-FromNAS updated (DEPLOY.BAT).
2026-03-20 Galactic Advisors security assessment — AD1 C: at 90%, legacy SQL 2008 R2 client noted, 3 computers scanned.
2026-03-23 Galactic Advisors assessment analyzed by ACG.
2026-03-27 Major security incident: DF-JOEL2 compromised via social engineering/ScreenConnect (attacker "Angel Raya", C2 on Virtuo hosting). M365 sign-in from Turkey. Full remediation. 3 CA policies deployed. MFA notice sent. IC3 filed (1c32ade367084be9acd548f23705736f).
2026-03-27–29 Test datasheet pipeline rebuilt — 72/73 Quatronix datasheets generated, new Node.js pipeline replaces VB6 DFWDS + VB.NET uploader.
2026-03-31 Joel Lohr retirement. Brian Faires mailbox converted to shared (5,711 messages preserved). 38 stale Entra TS-* accounts deleted.
2026-04-04 MFA CA policies enforced (switched from report-only).
2026-04-11–12 SCMVAS/SCMHVAS pipeline extension — 27,503 records backfilled, 434 Engineering-Tested .txt files imported.
2026-04-12 TestDataDB PostgreSQL migration verified (2.89M records). Hoffman API discovered (Swagger).
2026-04-13 API architecture discussion with Hoffman — client_credentials grant confirmed for dataforth.onprem.sync client.
2026-04-14 DFWDS logic ported to Node.js (dfwds-process.js). 897 staged datasheets drained. 803 new records created on Hoffman API.
2026-04-15 Major release — DB dedup (2.89M→469K rows), FAIL→PASS retest rule, For_Web filesystem dependency eliminated, 170,984 records bulk-pushed to Hoffman. Dashboard UI upgrades.
2026-04-23 Full Dataforth tenant onboarded to all 5 ComputerGuru tiered apps. calibration@ SMTP AUTH fixed. DF-GAGETRAK GuruRMM agent enrolled (with auth workaround). Syncro ticket #32142 billed.
2026-05-03 jantar@dataforth.com darkweb breach check — no indicators of compromise. eM Client OAuth grant and SP revoked/disabled. 1 hr billed.
2026-05-04 Howard onsite — lobby phone offline (VLAN misconfiguration on D1-Server-Room port 1 → fixed to VLAN 100).
2026-05-06 SAGE-SQL RDS issues resolved — grace period reset, SSL cert replaced, TSGateway disabled, RemoteApp permission prompts fixed.
2026-05-12 Pipeline audit + email notifications implemented (Graph API). jlohr forwarding configured (ntirety.com → mike@). DKIM keys rotated.
2026-06-01 AOI optical-inspection XP PC isolated onto VLAN 2 (mydata/SMT) @ 192.168.1.175; aoibackup SMB1 share created on D2TESTNAS locked to the XP only; other NAS shares set to deny the XP. D2TESTNAS confirmed Debian 13 / Samba 4.22.6 (repurposed Netgear ReadyNAS); vault + wiki OS corrected.
2026-06-01 Chauncey Bell (cbell) M365 verified — active mailbox, licensed M365 Business Standard; AD password reset on AD2 (synced user, OU=Azure_Users), signed into Office. Bobbi's Outlook printing fixed. Ticket #32364 (0.5 hr onsite).
2026-06-02 Syncro asset reconciliation (78 assets): 20 keep / 21 save+flag / 28 remove / 9 verify. Root cause identified: fleet-wide Syncro agent break ~2025-10-06 silenced ~half the fleet while boxes stayed online (visible in ScreenConnect). Dataforth confirmed phasing off Bitdefender. Cleanup list handed to Howard.
2026-06-04 SP1366 MAQ20 manufacturing print recovery — 19/20 PDFs for revisions E–H restored to AD2 from HGHAUBNER's pre-attack backup via GuruRMM user_session + GPO-mapped Q: drive. Root cause of loss: incomplete 10/1/2025 recovery restore. Syncro #32385, 1.0 hr remote, prepaid $0, resolved. GuruRMM fleet grew 13 → 45 agents. WizTree backup-side CSV captured for migration-gap diff (deferred).
2026-06-05 AD1 Files backup plan created via GuruRMM remote command (cbb.exe, NBF, 180-day retention, daily 2 AM, covers C:\Engineering + C:\Shares\ITSvc). AD1 now has both image and file plans matching AD2.
2026-06-05 Mailprotector CloudFilter discovered as Dataforth's outbound delivery layer (atop INKY + Exchange Online). Email from Georg Haubner was held by Mailprotector due to INKY "Annotation" transport rule. Released manually. New /mailprotector skill built and committed.
2026-06-05 Georg Haubner's Power Monitor SPA analyzed (vanilla-JS, AI-built). Gateway architecture designed for PWM.dataforth.com demo. Parked pending Mike↔Georg conversation.
2026-06-08–09 Total Dataforth phone outage. Outbound failed (FirstDigital SBC ignoring OPTIONS → trunk Unavailable); inbound never worked (no SIP port-forward existed). Fixed: qualify_frequency=0 in pjsip DB; PJSip.class.php line 504 re-patched; /data/on_boot.d/30-freepbx-sip-forward.sh added (SIP-only DNAT, source-locked 66.7.123.0/24). Two-way audio verified. UDM vault corrected. Syncro #32392, 1.0 hr emergency (×1.5 rate) remote, prepaid.
2026-06-10 Shares & Permissions Phase 0 complete. Read-only ACL audit of all 8 business shares: all grant Domain Users/Everyone Full or Modify; no department security groups exist; Payroll/OSHA/PO/accounting data open to all employees. Phase 1 (client input) pending discovery email to Dan Center.
2026-06-17 AD2 identity.json + Python 3.12.8 installed. CLAUDE.dataforth.md created for AD2 context file (relocated from in-line .claude/CLAUDE.md edits to maintain clean fork).
2026-06-18 DSCA33/45 certs recovered via Hoffman API — 56 model templates mined, 1,452 new DSCA33/45 certs published on AD2 (0 overwrites). Root-caused parseRawData bug affecting 8B/5B/SCM families. 136 8B/5B/SCM templates mined from Hoffman and handed to AD2 for wiring. TestDataDB UI redesigned and deployed on AD2 (cert-fit, publish chips, push toasts, full-screen inspector). AD2 SSH PMTU blackhole diagnosed (GURU-5070 adapter MTU 1500 vs tunnel ~1424) and fixed (MTU 1400). Syncro #32441.
2026-06-22 Shares & Permissions Phase 2 target-state strawman drafted — proposed Company\Departments\…Restricted\…Company-Wide\…Users\…Archive\ tree with `SG--<RW
2026-06-23 PBX no-inbound-calls emergency fixed. UDM SIP 5060 DNAT was completely absent — flushed by a UniFi controller provision (not a reboot), confirming the on_boot.d re-apply script's coverage gap is broader than previously understood. Re-ran /data/on_boot.d/30-freepbx-sip-forward.sh, verified DNAT + forward-accept restored, inbound confirmed working end-to-end. Vault pbx.sops.yaml password corrected (was backslash-corrupted from shell-escaping leaking into storage). Syncro #32450, 1.5 hr emergency (×1.5 rate) remote, invoiced, block debited 31.5 → 30.0 hrs. Durable-fix coord to-do 45572ee1 filed.
2026-06-24–25 DFORTH-Ship recurring BSOD diagnosed. Stop code 0x116 VIDEO_TDR_FAILURE on integrated Intel HD Graphics 4600 (final 2020 driver) — an 11.5-year-old HP EliteDesk 800 G1 USDT with an accelerating crash cadence (5 dumps since 2025-11-03). Mitigated by disabling Edge hardware acceleration via machine policy; PC replacement recommended as the durable fix; thermal cleaning flagged as a secondary measure.
2026-07-01 Test-data-chain ground-truth audit via a headless Claude spawned through AD2's GuruRMM agent (new capability, bypassing AD2's coord-API isolation for read-only investigation). Confirmed root cause of Syncro #32489: deployed NWTOC.BAT v5.0 never copies master spec .DAT files to stations (removed by design in the v5.0 changelog). New HIGH finding: NWTOC.BAT/CTONWTXT.BAT use COPY /Y, not a valid MS-DOS 6.22 switch — pending station-side DOS-version confirmation before scoping the fix (independently confirmed by Grok). Corrected several stale assumptions: datastore is PostgreSQL 18 (475,553 test_records), the sync task runs Sync-FromNAS-rsync.ps1, web delivery is a live HTTP API uploader (472,290 records flagged). Also found: a stray file breaking the AD2↔NAS rsync push on every run (masking real errors), a frozen 2026-03-27 server-side spec snapshot, and plaintext credentials hard-coded in three scripts (flagged for rotation, not yet fixed).
2026-07-04 MYDATA TPSys SMT controller (myserver, FC3/VLAN2) discovered + root recovered via LILO single-user; vaulted; RMM agent ruled out (legacy glibc/kernel/no-systemd).

Backlinks

  • projects/dataforth-dos — Active test datasheet pipeline project on AD2
  • systems/jupiter — Neptune Exchange physically colocated at Dataforth D2 facility; D2TESTNAS provides Tailscale routing
Reference in New Issue View Git Blame Copy Permalink