Files

OC-5070 b83d34ba50 Session log: Final update with AD1 session import, memory entries, MFA details

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-27 20:07:20 -07:00

17 KiB

Raw Permalink Blame History

Session Log: 2026-03-27 - Dataforth Security Incident, MFA Rollout, Test Datasheet Investigation

Session Summary

Major incident response and security hardening session at Dataforth Corporation. Three concurrent workstreams:

Security Incident - Compromised workstation DF-JOEL2 via social engineering / ScreenConnect abuse
MFA Rollout - Deployed Conditional Access policies for M365 tenant
Test Datasheet Pipeline - Investigated broken datasheet export pipeline affecting customer shipments

1. Security Incident: DF-JOEL2 Compromise

Timeline (March 27, 2026 MST)

08:25 - Joel Lohr clicked phishing link in personal Yahoo/Comcast email (appeared to be from Arizona Technology Council)
08:28 - ScreenConnect client installed from C:\Users\jlohr\Downloads\ScreenConnect.ClientSetup.msi
08:29 - "Angel Raya" connected via ScreenConnect cloud relay (instance-wlb9ga-relay.screenconnect.com)
08:29 - Attacker deployed two C2 backdoor ScreenConnect clients via PowerShell
08:31 - Attacker downloaded Sordum "Hide From Uninstall List" tool
08:32 - Tool used to hide rogue clients, "Angel Raya" disconnected
11:55 - "Administrator" connected via 80.76.49.18 C2 backdoor
12:40 - "Administrator" disconnected
~13:00 - ACG discovered compromise during ScreenConnect session
18:51 - Successful unauthorized M365 sign-in from Istanbul, Turkey

Attacker Infrastructure

C2 Server 1: 80.76.49.18:8040/8041
C2 Server 2: 45.88.91.99:8040/8041
ASN: AS399486, Virtuo (12651980 CANADA INC.), Montreal QC
Abuse: abuses@virtuo.host, escalation: jordan@virtuo.host
ScreenConnect Cloud: instance-wlb9ga-relay.screenconnect.com
C2 client version: 25.2.4.9229 (binaries dated April 8, 2025)

Rogue ScreenConnect Clients Found

0cad93610010625f - "Angel Raya" initial access (instance-wlb9ga cloud relay)
0dfe1abae029411c - C2 backdoor (80.76.49.18:8041)
a897d9a21259d116 - C2 backdoor (45.88.91.99:8041)
1912bf3444b41a08 - LEGITIMATE (ACG, instance-kgc7jt)

M365 Account Compromise

Sustained brute-force against jlohr@dataforth.com for 7+ days:

Successful: Istanbul Turkey (91.93.232.236), Croydon UK (82.44.33.210), Germany (IPv6)
Blocked attempts from: Frankfurt DE (45.86.202.x), Luxembourg, Virginia Beach, Sioux Falls, Camden DE, Charleston WV
Tools used: Azure AD PowerShell, Azure CLI
MFA: Password + phone only (520-917-2241), no conditional access

Remediation Actions Completed

[DONE] Both C2 IPs blocked at UDM firewall (iptables FORWARD + INPUT, all directions)
[DONE] Three rogue ScreenConnect clients uninstalled via WinRM
[DONE] HideUL tool deleted from C:\Users\Public\Pictures\Backup\
[DONE] Downloaded MSIs cleaned
[DONE] jlohr AD password reset to Dataforth2026! (force change at logon)
[DONE] Entra Connect delta sync forced
[DONE] M365 sessions revoked (twice)
[DONE] Network-wide scan: 32 machines clean, 28 unreachable (offline)
[DONE] UDM connection table scan: zero C2 traffic network-wide
[DONE] No malicious inbox rules, mail forwarding, or OAuth consents
[DONE] No rogue SSH keys on UDM

Reports Filed

FBI IC3: Submission ID 1c32ade367084be9acd548f23705736f (filed 3/27/2026 5:11 PM EST)
Virtuo Hosting: abuses@virtuo.host - automated suspension of both IPs confirmed
ConnectWise: Case #03464184 (abuse@connectwise.com)

Incident Notes File

PSA ticket notes: /tmp/dataforth-incident-psa-notes.txt
Virtuo abuse report: clients/dataforth/docs/incident-2026-03-27-abuse-report-virtuo.md
ConnectWise abuse report: clients/dataforth/docs/incident-2026-03-27-abuse-report-connectwise.md
IC3 complaint PDF: clients/dataforth/docs/IC3-Complaint-2026-03-27.pdf

2. MFA Rollout

Conditional Access Policies Deployed (Report-Only Mode)

Policy	ID	State
ACG - Require MFA for All Users	dc920ee4-22e6-402b-b5e3-4f3662d26227	Report-only
ACG - Block Foreign Sign-Ins	3405f7db-91b6-48da-b3fb-2e0ef1e44d17	Report-only
ACG - Block Legacy Authentication	82ebbe3b-d151-4cb7-aff7-af893a4915e3	Report-only

Named Locations

Dataforth Office - Tucson: 0a3e61d7-a544-4a47-961a-a98cd4804613 (67.206.163.122/32, trusted)
Allowed Countries - US Only: 12706cec-c91b-454e-a24d-c801284b79f7

Security Groups

MFA-Excluded-BreakGlass: 75ac10ae-d49e-42b1-aa87-04908a983495
- Members: Brian Faires, Dataforth Calibration, Dataforth Notifications, Endcap, Tablet 01
MFA-Travel-Bypass: 094b12c5-b39a-4287-943a-f1175ce61a6f
- For users traveling internationally, excludes them from foreign sign-in block

MFA policy behavior

From office (67.206.163.122): No MFA required
From elsewhere in US: MFA required
From outside US: Blocked (unless in MFA-Travel-Bypass group)
Legacy auth (IMAP/POP/old PowerShell): Blocked everywhere

Enforcement deadline: April 4, 2026

MFA notice email sent to all 37 licensed users from sysadmin@dataforth.com via Graph API (draft-then-send method)
Enrollment status email sent to mike@azcomputerguru.com and ghaubner@dataforth.com

MFA Enrollment Status (19 ready, 19 need setup)

Ready: Ben Wadzinski, Bobbi Whitson, Dan Center, Georg Haubner (Authenticator), Jacque Antar, Jaime Becerra, Joel Lohr, John Lehman, Kellyn Wackerly, Kevin Wackerly, Lee Payne, Otto Fest (Windows Hello), Peter Iliya, Robert Koranek, Sandra Schock, Shipping, sysadmin (Authenticator), Theresa Dean, Yvonne Bejarano

Need setup: Andres Oliva-Martinez, Angel Lopez, Ayleen Montijo, Bernardo Laredo, Catalina Vanatta, Cesar Rivas, Chauncey Bell, Concepcion Hernandez, Cynthia Roedig, Elma Trujillo, Jason Yoder, Ken Hoffman, Linda Duarte, Logan Tobey, Lori Schlotterback, Manny Vargas, Martin Florez, Rosalinda Duarte, Thomas Nord

Entra Cleanup

38 test station accounts (TS-*) deleted from Entra (were stale synced objects from CompanyUsers OU no longer in sync scope)
bfaires@dataforth.com: AD account disabled, M365 still enabled, mailbox converted to shared (5,711 messages preserved)

Dataforth M365 Licenses

50x M365 Business Premium (39 used) - includes Entra ID P1
19x Exchange Online Plan 1 (5 used)
5x SPB (4 used)

3. Test Datasheet Pipeline Investigation

Background

Customer Quatronix (China) refusing shipments of 54+ modules without test datasheets. Originally 328 missing, whittled to 54 by Peter Iliya manually finding some.

Pipeline Architecture

DOS Test Machine (TS-XX) -> QuickBASIC test program
  -> Generates H-prefix TXT file (H=17 decode: A=10,B=11...H=17,I=18,J=19)
  -> Writes to T:\ (mapped to \\D2TESTNAS\TEST)
  -> Syncs to AD2 C:\Shares\test\ (Sync-FromNAS task, every 15 min, WORKING)
  -> TestDataDB import.js ingests DAT files into SQLite database (WORKING)
  -> DFWDS.exe should process and move to WebShare (BROKEN - third party, Hoffman)
  -> TestDataSheetUploader should sync WebShare to website (BROKEN - not running since 2022)
  -> Customer downloads from www.dataforth.com

Current State

Sync-FromNAS: RUNNING (last ran 5:00 PM today, every 15 min)
TestDataDB service: RUNNING (Windows service, auto-start, 2.27M records)
TestDataDB API: http://192.168.0.6:3000 (Express.js, SQLite, better-sqlite3)
DFWDS.exe: NOT RUNNING (VB6 program on AD2, third-party dev Hoffman unresponsive)
TestDataSheetUploader: NOT RUNNING (last used Nov 2022, config points to Hoffman's local path)
datasheet_exported_at: NULL for ALL records - export has never run

Key Finding

Of the missing Quatronix serials checked, 22 out of 22 that responded are IN the database with test data. Zero actually missing. The data exists but has never been exported as datasheet files.

TestDataSheetUploader Details (found in Test Datasheets folder)

VB.NET console app, .NET Framework 4.7.2
Services: https://www.dataforth.com/Services/{Uploader,DirectoryManifest,DeleteFile}.aspx
Auth: DataforthWebShare / Data6277
Server-side path: C:\inetpub\wwwroot\dataforth.com\WebShare\ProcessCheckOutTestFolder\
Config: C:\Users\hoffm\Documents\Customer Folders\Dataforth\product lists (Hoffman's old path)

WebShare on AD2

Share: \AD2\webshare -> C:\Shares\webshare
DFWDS config expects: X:\webshare\Test_Datasheets, X:\webshare\For_Web, X:\webshare\Bad_Datasheets
DFWDS/Test Files/Original/Test_Datasheets: 2,457 existing datasheets

H-prefix decode table

A=10, B=11, C=12, D=13, E=14, F=15, G=16, H=17, I=18, J=19 Example: H8601-6.TXT -> 178601-6.TXT

AD1 Claude Instance Results (same day)

A Claude instance running on AD1/AD2 rebuilt the entire pipeline:

Spec Parser built (parsers/spec-reader.js) - reads 8 binary spec DATs, 1470 models
Exact-match TXT formatter (templates/datasheet-exact.js) - reverse-engineered from QB source
Auto-export system (database/export-datasheets.js) - generates TXT to X:\For_Web after import
72 of 73 Quatronix datasheets generated - in ComputerGuruConnectv2/Files/Quatronix-Datasheets.zip
1.4M records reconciled with existing For_Web files, 500K archived to year subfolders
Root cause: CTONWTXT.BAT not called in AUTOEXEC v4.1 since 2026-03-12. TXT piling up in C:\STAGE since Sept 2025.
New pipeline eliminates CTONWTXT.BAT, DFWDS.exe, and TestDataSheetUploader
1 missing datasheet: 177000-15 (SCM5B49-05) needs empty spec file from John Lehman
Website upload still broken: old ASP.NET endpoints return 404, need replacement
7B series: ~830K records need 7B-specific formatter (specs loaded but layout not implemented)
All session files saved to clients/dataforth/session-logs/ (from ClaudeSession-2026-03-27.zip)

Next Steps for Datasheets

Website upload replacement (old endpoints dead, need new mechanism)
Fix CTONWTXT.BAT in AUTOEXEC or accept server-side generation as sufficient
7B datasheet formatting
SCM5B49 spec file from John
New product lines: MAQ20/PWRM (XLS), 10D (JSON, ~May 2026)

4. Other Work

UDM Firewall Access

Generated SSH key: ~/.ssh/id_ed25519_udm
Public key added to UDM root authorized_keys
UDM IPs: 192.168.0.254 (also responds on 192.168.0.1, same MAC d0:21:f9:6c:11:02)

Network Investigation (Internet/Phone Outage Report)

All infrastructure reachable: UDM, AD1, AD2, D2TESTNAS
Internet working from NAS (8.8.8.8, google.com)
DNS working on all DCs and UDM
ISP: fdtnet.net, Dataforth public IP: 67.206.163.122
Issue was likely localized to specific machines or resolved itself

AD1 Claude Instance README

Created: clients/dataforth/dos-test-machines/Test Datasheets/README.md
Contains Gitea credentials, network context, investigation steps for AD1 instance

Credentials Used This Session

Dataforth Network

AD1 (192.168.0.27): INTRANET\sysadmin / Paper123!@#
AD2 (192.168.0.6): INTRANET\sysadmin / Paper123!@#
D2TESTNAS (192.168.0.9): root / SSH key (~/.ssh/id_ed25519 default)
UDM (192.168.0.254): root / SSH key (~/.ssh/id_ed25519_udm)

Dataforth M365

Tenant ID: 7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584
Admin: sysadmin@dataforth.com / Paper123!@# (synced with AD)
Entra App (Claude-Code-M365): App ID 7a8c0b2e-57fb-4d79-9b5a-4b88d21b1f29 / Secret tXo8Q~ZNG9zoBpbK9HwJTkzx.YEigZ9AynoSrca3

MSP Multi-Tenant App (Claude-MSP-Access)

MSP Tenant: ce61461e-81a0-4c84-bb4a-7b354a9a356d
App ID: fabb3421-8b34-484b-bc17-e46de9703418
Client Secret: ~~QJ8Q~~NyQSs4OcGqHZyPrA2CVnq9KBfKiimntbMO
Permissions granted for Dataforth tenant: CA policies, mail, users, auth methods, security events

TestDataSheetUploader Web Services

URL: https://www.dataforth.com/Services/{Uploader,DirectoryManifest,DeleteFile}.aspx
Auth: DataforthWebShare / Data6277

Rsync (Sync-FromNAS on AD2)

NAS rsync user: rsync / IQ203s32119
Module: test (maps to /data/test on NAS)

Gitea

URL: https://git.azcomputerguru.com
User: mike@azcomputerguru.com
Password: Gptf*77ttb123!@#-git

WinRM Python Environment

/tmp/winrm-env/bin/python3 (pywinrm, msal, requests installed)

Outstanding / Incomplete

Security

28 machines unreachable during ScreenConnect scan - check when online
Joel should reset Yahoo/Comcast personal email password
Check Chrome saved passwords on DF-JOEL2 (may have been exported by attacker)
Fix Windows Defender on DF-JOEL2 (error 0x800106ba, McAfee conflict)
Add C2 IPs to UDM permanent block list via UniFi UI (iptables rules don't survive reboot)
Monitor ConnectWise case 03464184 for Angel Raya identity
Consider disabling jlohr account entirely (retiring March 31)
Original phishing email never found (sent to personal Yahoo, Joel can't locate it)

MFA

April 4: Switch all 3 CA policies from report-only to enabled
19 users still need to register MFA
Delete orphan scheduled tasks on AD2: TestDataDB Server, TestDataDB_NodeServer
Clean up remaining unlicensed enabled accounts in Entra (former employees)

Test Datasheets

[DONE] Pipeline rebuilt by AD1 instance -- server-side TXT generation from DAT data
[DONE] 72 of 73 Quatronix datasheets generated (in ComputerGuruConnectv2/Files/Quatronix-Datasheets.zip)
1 remaining: 177000-15 (SCM5B49-05) needs empty spec file from John Lehman
Website upload replacement -- old ASP.NET endpoints return 404
7B datasheet formatting (~830K records, specs loaded but formatter needs 7B layout)
Fix CTONWTXT.BAT in AUTOEXEC or accept server-side generation as sufficient
New product lines: MAQ20/PWRM (XLS), 10D (JSON, ~May 2026)
Service permissions cleanup (runs as SYSTEM)

Update: 17:45 - AD1 Session Import and Final Context Save

AD1 Claude Instance Session (same day)

Imported full session from /home/guru/ComputerGuruConnectv2/Files/ClaudeSession-2026-03-27.zip. The AD1 instance running on AD2 rebuilt the entire test datasheet pipeline:

Key accomplishments:

Built spec parser (parsers/spec-reader.js) - reads 8 binary spec DATs, 1470 models across 6 families
Built exact-match TXT formatter (templates/datasheet-exact.js) - reverse-engineered from QuickBASIC source
Built auto-export system (database/export-datasheets.js) - generates TXT to X:\For_Web after import
Added PDF on-demand endpoint (/api/datasheet/:id/pdf)
Generated 72 of 73 Quatronix datasheets
Reconciled 1.4M records with existing For_Web files
Archived 500K pre-2026 files into year subfolders
Found root cause: CTONWTXT.BAT not called in AUTOEXEC v4.1 since 2026-03-12

AD1 session files saved to:

clients/dataforth/session-logs/SESSION-SUMMARY.md
clients/dataforth/session-logs/project_pipeline_rebuilt.md
clients/dataforth/session-logs/project_test_datasheet_pipeline.md
clients/dataforth/session-logs/project_ctonwtxt_gap.md
clients/dataforth/session-logs/project_ad2_context.md
clients/dataforth/session-logs/project_engr_share_exploration.md
clients/dataforth/session-logs/project_new_product_lines.md
clients/dataforth/session-logs/reference_haubner_backup.md
clients/dataforth/session-logs/feedback_sn_from_data.md
clients/dataforth/session-logs/email-draft-updated.md
clients/dataforth/session-logs/email-draft-morning.md
clients/dataforth/session-logs/user_mike_swanson.md
clients/dataforth/session-logs/MEMORY.md (AD1 instance memory index)

Memory Entries Created

.claude/memory/project_datasheet_pipeline.md - Full pipeline architecture and status
.claude/memory/project_dataforth_incident_2026-03-27.md - Security incident and MFA summary
MEMORY.md index updated with both entries

MFA Note

SMS/Voice calling acceptable as MFA method for users who push back on Authenticator app
CA policy requires MFA but doesn't mandate specific method - phone registration is sufficient
MFA-Travel-Bypass group (094b12c5) created for users traveling internationally

Emails Sent via Graph API (draft-then-send from sysadmin@dataforth.com)

MFA notice to all 37 licensed users (deadline April 4, 2026)
Enrollment status report to mike@azcomputerguru.com and ghaubner@dataforth.com

Gitea

502 error on push - Gitea container may need restart on Jupiter
All changes committed locally, will push when available

Files Created/Modified This Session

clients/dataforth/dos-test-machines/Test Datasheets/README.md (AD1 Claude instance context)
clients/dataforth/docs/incident-2026-03-27-abuse-report-virtuo.md
clients/dataforth/docs/incident-2026-03-27-abuse-report-connectwise.md
clients/dataforth/docs/IC3-Complaint-2026-03-27.pdf
.claude/memory/project_datasheet_pipeline.md
.claude/memory/project_dataforth_incident_2026-03-27.md
.claude/memory/MEMORY.md (updated index)
clients/dataforth/session-logs/* (13 AD1 session files imported)
/tmp/dataforth-incident-psa-notes.txt (PSA ticket notes)
/tmp/dataforth-mfa-notice.txt (MFA user communication)
/tmp/ic3-technical-details.txt (IC3 technical details)
~/.ssh/id_ed25519_udm (SSH key for UDM access)
/tmp/winrm-env/ (Python venv with pywinrm, msal, requests)

17 KiB Raw Permalink Blame History

Session Log: 2026-03-27 - Dataforth Security Incident, MFA Rollout, Test Datasheet Investigation

Session Summary

1. Security Incident: DF-JOEL2 Compromise

Timeline (March 27, 2026 MST)

Attacker Infrastructure

Rogue ScreenConnect Clients Found

M365 Account Compromise

Remediation Actions Completed

Reports Filed

Incident Notes File

2. MFA Rollout

Conditional Access Policies Deployed (Report-Only Mode)

Named Locations

Security Groups

MFA policy behavior

Enforcement deadline: April 4, 2026

MFA Enrollment Status (19 ready, 19 need setup)

Entra Cleanup

Dataforth M365 Licenses

3. Test Datasheet Pipeline Investigation

Background

Pipeline Architecture

Current State

Key Finding

TestDataSheetUploader Details (found in Test Datasheets folder)

WebShare on AD2

H-prefix decode table

AD1 Claude Instance Results (same day)

Next Steps for Datasheets

4. Other Work

UDM Firewall Access

Network Investigation (Internet/Phone Outage Report)

AD1 Claude Instance README

Credentials Used This Session

Dataforth Network

Dataforth M365

MSP Multi-Tenant App (Claude-MSP-Access)

TestDataSheetUploader Web Services

Rsync (Sync-FromNAS on AD2)

Gitea

WinRM Python Environment

Outstanding / Incomplete

Security

MFA

Test Datasheets

Update: 17:45 - AD1 Session Import and Final Context Save

AD1 Claude Instance Session (same day)

Memory Entries Created

MFA Note

Emails Sent via Graph API (draft-then-send from sysadmin@dataforth.com)

Gitea

Files Created/Modified This Session

17 KiB

Raw Permalink Blame History