Session log: Dataforth security incident, MFA rollout, test datasheet investigation
- DF-JOEL2 compromised via ScreenConnect social engineering (Angel Raya) - C2 IPs blocked, rogue clients removed, M365 sessions revoked, password reset - IC3 complaint filed, abuse reports sent to Virtuo and ConnectWise - Conditional Access policies deployed (MFA, block foreign, block legacy auth) - 38 stale test station accounts deleted from Entra - Test datasheet pipeline investigated - data exists in DB, export step broken - TestDataSheetUploader source code extracted for analysis Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
OC-5070
@@ -0,0 +1,263 @@
|
||||
# Session Log: 2026-03-27 - Dataforth Security Incident, MFA Rollout, Test Datasheet Investigation
|
||||
|
||||
## Session Summary
|
||||
|
||||
Major incident response and security hardening session at Dataforth Corporation. Three concurrent workstreams:
|
||||
|
||||
1. **Security Incident** - Compromised workstation DF-JOEL2 via social engineering / ScreenConnect abuse
|
||||
2. **MFA Rollout** - Deployed Conditional Access policies for M365 tenant
|
||||
3. **Test Datasheet Pipeline** - Investigated broken datasheet export pipeline affecting customer shipments
|
||||
|
||||
---
|
||||
|
||||
## 1. Security Incident: DF-JOEL2 Compromise
|
||||
|
||||
### Timeline (March 27, 2026 MST)
|
||||
- 08:25 - Joel Lohr clicked phishing link in personal Yahoo/Comcast email (appeared to be from Arizona Technology Council)
|
||||
- 08:28 - ScreenConnect client installed from C:\Users\jlohr\Downloads\ScreenConnect.ClientSetup.msi
|
||||
- 08:29 - "Angel Raya" connected via ScreenConnect cloud relay (instance-wlb9ga-relay.screenconnect.com)
|
||||
- 08:29 - Attacker deployed two C2 backdoor ScreenConnect clients via PowerShell
|
||||
- 08:31 - Attacker downloaded Sordum "Hide From Uninstall List" tool
|
||||
- 08:32 - Tool used to hide rogue clients, "Angel Raya" disconnected
|
||||
- 11:55 - "Administrator" connected via 80.76.49.18 C2 backdoor
|
||||
- 12:40 - "Administrator" disconnected
|
||||
- ~13:00 - ACG discovered compromise during ScreenConnect session
|
||||
- 18:51 - Successful unauthorized M365 sign-in from Istanbul, Turkey
|
||||
|
||||
### Attacker Infrastructure
|
||||
- C2 Server 1: 80.76.49.18:8040/8041
|
||||
- C2 Server 2: 45.88.91.99:8040/8041
|
||||
- ASN: AS399486, Virtuo (12651980 CANADA INC.), Montreal QC
|
||||
- Abuse: abuses@virtuo.host, escalation: jordan@virtuo.host
|
||||
- ScreenConnect Cloud: instance-wlb9ga-relay.screenconnect.com
|
||||
- C2 client version: 25.2.4.9229 (binaries dated April 8, 2025)
|
||||
|
||||
### Rogue ScreenConnect Clients Found
|
||||
1. 0cad93610010625f - "Angel Raya" initial access (instance-wlb9ga cloud relay)
|
||||
2. 0dfe1abae029411c - C2 backdoor (80.76.49.18:8041)
|
||||
3. a897d9a21259d116 - C2 backdoor (45.88.91.99:8041)
|
||||
4. 1912bf3444b41a08 - LEGITIMATE (ACG, instance-kgc7jt)
|
||||
|
||||
### M365 Account Compromise
|
||||
Sustained brute-force against jlohr@dataforth.com for 7+ days:
|
||||
- Successful: Istanbul Turkey (91.93.232.236), Croydon UK (82.44.33.210), Germany (IPv6)
|
||||
- Blocked attempts from: Frankfurt DE (45.86.202.x), Luxembourg, Virginia Beach, Sioux Falls, Camden DE, Charleston WV
|
||||
- Tools used: Azure AD PowerShell, Azure CLI
|
||||
- MFA: Password + phone only (520-917-2241), no conditional access
|
||||
|
||||
### Remediation Actions Completed
|
||||
- [DONE] Both C2 IPs blocked at UDM firewall (iptables FORWARD + INPUT, all directions)
|
||||
- [DONE] Three rogue ScreenConnect clients uninstalled via WinRM
|
||||
- [DONE] HideUL tool deleted from C:\Users\Public\Pictures\Backup\
|
||||
- [DONE] Downloaded MSIs cleaned
|
||||
- [DONE] jlohr AD password reset to Dataforth2026! (force change at logon)
|
||||
- [DONE] Entra Connect delta sync forced
|
||||
- [DONE] M365 sessions revoked (twice)
|
||||
- [DONE] Network-wide scan: 32 machines clean, 28 unreachable (offline)
|
||||
- [DONE] UDM connection table scan: zero C2 traffic network-wide
|
||||
- [DONE] No malicious inbox rules, mail forwarding, or OAuth consents
|
||||
- [DONE] No rogue SSH keys on UDM
|
||||
|
||||
### Reports Filed
|
||||
- FBI IC3: Submission ID 1c32ade367084be9acd548f23705736f (filed 3/27/2026 5:11 PM EST)
|
||||
- Virtuo Hosting: abuses@virtuo.host - automated suspension of both IPs confirmed
|
||||
- ConnectWise: Case #03464184 (abuse@connectwise.com)
|
||||
|
||||
### Incident Notes File
|
||||
- PSA ticket notes: /tmp/dataforth-incident-psa-notes.txt
|
||||
- Virtuo abuse report: clients/dataforth/docs/incident-2026-03-27-abuse-report-virtuo.md
|
||||
- ConnectWise abuse report: clients/dataforth/docs/incident-2026-03-27-abuse-report-connectwise.md
|
||||
- IC3 complaint PDF: clients/dataforth/docs/IC3-Complaint-2026-03-27.pdf
|
||||
|
||||
---
|
||||
|
||||
## 2. MFA Rollout
|
||||
|
||||
### Conditional Access Policies Deployed (Report-Only Mode)
|
||||
|
||||
| Policy | ID | State |
|
||||
|--------|-----|-------|
|
||||
| ACG - Require MFA for All Users | dc920ee4-22e6-402b-b5e3-4f3662d26227 | Report-only |
|
||||
| ACG - Block Foreign Sign-Ins | 3405f7db-91b6-48da-b3fb-2e0ef1e44d17 | Report-only |
|
||||
| ACG - Block Legacy Authentication | 82ebbe3b-d151-4cb7-aff7-af893a4915e3 | Report-only |
|
||||
|
||||
### Named Locations
|
||||
- Dataforth Office - Tucson: 0a3e61d7-a544-4a47-961a-a98cd4804613 (67.206.163.122/32, trusted)
|
||||
- Allowed Countries - US Only: 12706cec-c91b-454e-a24d-c801284b79f7
|
||||
|
||||
### Security Groups
|
||||
- MFA-Excluded-BreakGlass: 75ac10ae-d49e-42b1-aa87-04908a983495
|
||||
- Members: Brian Faires, Dataforth Calibration, Dataforth Notifications, Endcap, Tablet 01
|
||||
- MFA-Travel-Bypass: 094b12c5-b39a-4287-943a-f1175ce61a6f
|
||||
- For users traveling internationally, excludes them from foreign sign-in block
|
||||
|
||||
### MFA policy behavior
|
||||
- From office (67.206.163.122): No MFA required
|
||||
- From elsewhere in US: MFA required
|
||||
- From outside US: Blocked (unless in MFA-Travel-Bypass group)
|
||||
- Legacy auth (IMAP/POP/old PowerShell): Blocked everywhere
|
||||
|
||||
### Enforcement deadline: April 4, 2026
|
||||
- MFA notice email sent to all 37 licensed users from sysadmin@dataforth.com via Graph API (draft-then-send method)
|
||||
- Enrollment status email sent to mike@azcomputerguru.com and ghaubner@dataforth.com
|
||||
|
||||
### MFA Enrollment Status (19 ready, 19 need setup)
|
||||
**Ready:** Ben Wadzinski, Bobbi Whitson, Dan Center, Georg Haubner (Authenticator), Jacque Antar, Jaime Becerra, Joel Lohr, John Lehman, Kellyn Wackerly, Kevin Wackerly, Lee Payne, Otto Fest (Windows Hello), Peter Iliya, Robert Koranek, Sandra Schock, Shipping, sysadmin (Authenticator), Theresa Dean, Yvonne Bejarano
|
||||
|
||||
**Need setup:** Andres Oliva-Martinez, Angel Lopez, Ayleen Montijo, Bernardo Laredo, Catalina Vanatta, Cesar Rivas, Chauncey Bell, Concepcion Hernandez, Cynthia Roedig, Elma Trujillo, Jason Yoder, Ken Hoffman, Linda Duarte, Logan Tobey, Lori Schlotterback, Manny Vargas, Martin Florez, Rosalinda Duarte, Thomas Nord
|
||||
|
||||
### Entra Cleanup
|
||||
- 38 test station accounts (TS-*) deleted from Entra (were stale synced objects from CompanyUsers OU no longer in sync scope)
|
||||
- bfaires@dataforth.com: AD account disabled, M365 still enabled, mailbox converted to shared (5,711 messages preserved)
|
||||
|
||||
### Dataforth M365 Licenses
|
||||
- 50x M365 Business Premium (39 used) - includes Entra ID P1
|
||||
- 19x Exchange Online Plan 1 (5 used)
|
||||
- 5x SPB (4 used)
|
||||
|
||||
---
|
||||
|
||||
## 3. Test Datasheet Pipeline Investigation
|
||||
|
||||
### Background
|
||||
Customer Quatronix (China) refusing shipments of 54+ modules without test datasheets. Originally 328 missing, whittled to 54 by Peter Iliya manually finding some.
|
||||
|
||||
### Pipeline Architecture
|
||||
```
|
||||
DOS Test Machine (TS-XX) -> QuickBASIC test program
|
||||
-> Generates H-prefix TXT file (H=17 decode: A=10,B=11...H=17,I=18,J=19)
|
||||
-> Writes to T:\ (mapped to \\D2TESTNAS\TEST)
|
||||
-> Syncs to AD2 C:\Shares\test\ (Sync-FromNAS task, every 15 min, WORKING)
|
||||
-> TestDataDB import.js ingests DAT files into SQLite database (WORKING)
|
||||
-> DFWDS.exe should process and move to WebShare (BROKEN - third party, Hoffman)
|
||||
-> TestDataSheetUploader should sync WebShare to website (BROKEN - not running since 2022)
|
||||
-> Customer downloads from www.dataforth.com
|
||||
```
|
||||
|
||||
### Current State
|
||||
- Sync-FromNAS: RUNNING (last ran 5:00 PM today, every 15 min)
|
||||
- TestDataDB service: RUNNING (Windows service, auto-start, 2.27M records)
|
||||
- TestDataDB API: http://192.168.0.6:3000 (Express.js, SQLite, better-sqlite3)
|
||||
- DFWDS.exe: NOT RUNNING (VB6 program on AD2, third-party dev Hoffman unresponsive)
|
||||
- TestDataSheetUploader: NOT RUNNING (last used Nov 2022, config points to Hoffman's local path)
|
||||
- datasheet_exported_at: NULL for ALL records - export has never run
|
||||
|
||||
### Key Finding
|
||||
Of the missing Quatronix serials checked, **22 out of 22 that responded are IN the database** with test data. Zero actually missing. The data exists but has never been exported as datasheet files.
|
||||
|
||||
### TestDataSheetUploader Details (found in Test Datasheets folder)
|
||||
- VB.NET console app, .NET Framework 4.7.2
|
||||
- Services: https://www.dataforth.com/Services/{Uploader,DirectoryManifest,DeleteFile}.aspx
|
||||
- Auth: DataforthWebShare / Data6277
|
||||
- Server-side path: C:\inetpub\wwwroot\dataforth.com\WebShare\ProcessCheckOutTestFolder\
|
||||
- Config: C:\Users\hoffm\Documents\Customer Folders\Dataforth\product lists (Hoffman's old path)
|
||||
|
||||
### WebShare on AD2
|
||||
- Share: \\AD2\webshare -> C:\Shares\webshare
|
||||
- DFWDS config expects: X:\webshare\Test_Datasheets, X:\webshare\For_Web, X:\webshare\Bad_Datasheets
|
||||
- DFWDS/Test Files/Original/Test_Datasheets: 2,457 existing datasheets
|
||||
|
||||
### H-prefix decode table
|
||||
A=10, B=11, C=12, D=13, E=14, F=15, G=16, H=17, I=18, J=19
|
||||
Example: H8601-6.TXT -> 178601-6.TXT
|
||||
|
||||
### Next Steps for Datasheets
|
||||
- Need to build export mechanism to generate TXT datasheets from database records
|
||||
- TXT format matches the H8601-6.TXT sample (standard Dataforth test data sheet format)
|
||||
- Files need to land in C:\Shares\webshare\Test_Datasheets on AD2
|
||||
- Then need mechanism to get them to the website (either fix TestDataSheetUploader or replace it)
|
||||
|
||||
---
|
||||
|
||||
## 4. Other Work
|
||||
|
||||
### UDM Firewall Access
|
||||
- Generated SSH key: ~/.ssh/id_ed25519_udm
|
||||
- Public key added to UDM root authorized_keys
|
||||
- UDM IPs: 192.168.0.254 (also responds on 192.168.0.1, same MAC d0:21:f9:6c:11:02)
|
||||
|
||||
### Network Investigation (Internet/Phone Outage Report)
|
||||
- All infrastructure reachable: UDM, AD1, AD2, D2TESTNAS
|
||||
- Internet working from NAS (8.8.8.8, google.com)
|
||||
- DNS working on all DCs and UDM
|
||||
- ISP: fdtnet.net, Dataforth public IP: 67.206.163.122
|
||||
- Issue was likely localized to specific machines or resolved itself
|
||||
|
||||
### AD1 Claude Instance README
|
||||
- Created: clients/dataforth/dos-test-machines/Test Datasheets/README.md
|
||||
- Contains Gitea credentials, network context, investigation steps for AD1 instance
|
||||
|
||||
---
|
||||
|
||||
## Credentials Used This Session
|
||||
|
||||
### Dataforth Network
|
||||
- AD1 (192.168.0.27): INTRANET\sysadmin / Paper123!@#
|
||||
- AD2 (192.168.0.6): INTRANET\sysadmin / Paper123!@#
|
||||
- D2TESTNAS (192.168.0.9): root / SSH key (~/.ssh/id_ed25519 default)
|
||||
- UDM (192.168.0.254): root / SSH key (~/.ssh/id_ed25519_udm)
|
||||
|
||||
### Dataforth M365
|
||||
- Tenant ID: 7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584
|
||||
- Admin: sysadmin@dataforth.com / Paper123!@# (synced with AD)
|
||||
- Entra App (Claude-Code-M365): App ID 7a8c0b2e-57fb-4d79-9b5a-4b88d21b1f29 / Secret tXo8Q~ZNG9zoBpbK9HwJTkzx.YEigZ9AynoSrca3
|
||||
|
||||
### MSP Multi-Tenant App (Claude-MSP-Access)
|
||||
- MSP Tenant: ce61461e-81a0-4c84-bb4a-7b354a9a356d
|
||||
- App ID: fabb3421-8b34-484b-bc17-e46de9703418
|
||||
- Client Secret: ~QJ8Q~NyQSs4OcGqHZyPrA2CVnq9KBfKiimntbMO
|
||||
- Permissions granted for Dataforth tenant: CA policies, mail, users, auth methods, security events
|
||||
|
||||
### TestDataSheetUploader Web Services
|
||||
- URL: https://www.dataforth.com/Services/{Uploader,DirectoryManifest,DeleteFile}.aspx
|
||||
- Auth: DataforthWebShare / Data6277
|
||||
|
||||
### Rsync (Sync-FromNAS on AD2)
|
||||
- NAS rsync user: rsync / IQ203s32119
|
||||
- Module: test (maps to /data/test on NAS)
|
||||
|
||||
### Gitea
|
||||
- URL: https://git.azcomputerguru.com
|
||||
- User: mike@azcomputerguru.com
|
||||
- Password: Gptf*77ttb123!@#-git
|
||||
|
||||
### WinRM Python Environment
|
||||
- /tmp/winrm-env/bin/python3 (pywinrm, msal, requests installed)
|
||||
|
||||
---
|
||||
|
||||
## Outstanding / Incomplete
|
||||
|
||||
### Security
|
||||
- [ ] 28 machines unreachable during ScreenConnect scan - check when online
|
||||
- [ ] Joel should reset Yahoo/Comcast personal email password
|
||||
- [ ] Check Chrome saved passwords on DF-JOEL2 (may have been exported by attacker)
|
||||
- [ ] Fix Windows Defender on DF-JOEL2 (error 0x800106ba, McAfee conflict)
|
||||
- [ ] Add C2 IPs to UDM permanent block list via UniFi UI (iptables rules don't survive reboot)
|
||||
- [ ] Monitor ConnectWise case 03464184 for Angel Raya identity
|
||||
- [ ] Consider disabling jlohr account entirely (retiring March 31)
|
||||
- [ ] Original phishing email never found (sent to personal Yahoo, Joel can't locate it)
|
||||
|
||||
### MFA
|
||||
- [ ] April 4: Switch all 3 CA policies from report-only to enabled
|
||||
- [ ] 19 users still need to register MFA
|
||||
- [ ] Delete orphan scheduled tasks on AD2: TestDataDB Server, TestDataDB_NodeServer
|
||||
- [ ] Clean up remaining unlicensed enabled accounts in Entra (former employees)
|
||||
|
||||
### Test Datasheets
|
||||
- [ ] Build datasheet export mechanism from TestDataDB to WebShare
|
||||
- [ ] Either fix TestDataSheetUploader or build replacement for WebShare -> website upload
|
||||
- [ ] Resolve 54 missing datasheets for Quatronix customer (data IS in DB, needs export)
|
||||
- [ ] Contact Hoffman about DFWDS.exe status (third party, unresponsive)
|
||||
|
||||
---
|
||||
|
||||
## Files Created/Modified This Session
|
||||
- clients/dataforth/dos-test-machines/Test Datasheets/README.md (AD1 Claude instance context)
|
||||
- clients/dataforth/docs/incident-2026-03-27-abuse-report-virtuo.md
|
||||
- clients/dataforth/docs/incident-2026-03-27-abuse-report-connectwise.md
|
||||
- clients/dataforth/docs/IC3-Complaint-2026-03-27.pdf
|
||||
- /tmp/dataforth-incident-psa-notes.txt (PSA ticket notes)
|
||||
- /tmp/dataforth-mfa-notice.txt (MFA user communication)
|
||||
- /tmp/ic3-technical-details.txt (IC3 technical details)
|
||||
- ~/.ssh/id_ed25519_udm (SSH key for UDM access)
|
||||
Reference in New Issue
Block a user