584 lines
60 KiB
Markdown
584 lines
60 KiB
Markdown
---
|
||
type: client
|
||
name: dataforth
|
||
display_name: Dataforth Corporation
|
||
last_compiled: 2026-07-14
|
||
compiled_by: HOWARD-HOME/claude-main
|
||
sources:
|
||
- clients/dataforth/docs/overview.md
|
||
- clients/dataforth/docs/active-directory.md
|
||
- clients/dataforth/docs/workstations.md
|
||
- clients/dataforth/docs/manufacturing.md
|
||
- clients/dataforth/docs/billing-log.md
|
||
- clients/dataforth/docs/SYNC_SCRIPT_UPDATE_SUMMARY.md
|
||
- clients/dataforth/docs/incident-2026-03-27-abuse-report-virtuo.md
|
||
- clients/dataforth/docs/incident-2026-03-27-abuse-report-connectwise.md
|
||
- clients/dataforth/docs/cloud/m365.md
|
||
- clients/dataforth/docs/issues/log.md
|
||
- clients/dataforth/docs/network/topology.md
|
||
- clients/dataforth/docs/network/vlans.md
|
||
- clients/dataforth/docs/network/firewall.md
|
||
- clients/dataforth/docs/rmm/rmm.md
|
||
- clients/dataforth/docs/security/antivirus.md
|
||
- clients/dataforth/docs/security/backup.md
|
||
- clients/dataforth/docs/servers/ad1.md
|
||
- clients/dataforth/docs/servers/ad2.md
|
||
- clients/dataforth/docs/servers/d2testnas.md
|
||
- clients/dataforth/docs/servers/df-hyperv-b.md
|
||
- clients/dataforth/docs/servers/files-d1.md
|
||
- clients/dataforth/docs/servers/sage-sql.md
|
||
- clients/dataforth/docs/projects/shares-permissions/roadmap.md
|
||
- clients/dataforth/docs/projects/shares-permissions/current-state-2026-06-10.md
|
||
- clients/dataforth/docs/projects/shares-permissions/acl-audit-detail-2026-06-10.md
|
||
- clients/dataforth/docs/projects/shares-permissions/discovery-email-draft.md
|
||
- clients/dataforth/docs/projects/shares-permissions/target-structure-draft-2026-06-22.md
|
||
- clients/dataforth/docs/aoi-xp-vlan-backup-runbook.md
|
||
- clients/dataforth/session-logs/2026-03-23-galactic-advisors-report.md
|
||
- clients/dataforth/session-logs/2026-03-27-security-incident-mfa-datasheets.md
|
||
- clients/dataforth/session-logs/SESSION-SUMMARY.md
|
||
- clients/dataforth/session-logs/MEMORY.md
|
||
- clients/dataforth/session-logs/2026-04-12-session.md
|
||
- clients/dataforth/session-logs/2026-04-13-session.md
|
||
- clients/dataforth/session-logs/2026-04-14-session.md
|
||
- clients/dataforth/session-logs/2026-04-23-session.md
|
||
- clients/dataforth/session-logs/2026-05-03-session.md
|
||
- clients/dataforth/session-logs/2026-05-04-lobby-phone-vlan-fix.md
|
||
- clients/dataforth/session-logs/2026-05-06-session.md
|
||
- clients/dataforth/session-logs/2026-05-12-session.md
|
||
- clients/dataforth/session-logs/2026-06-01-aoi-xp-vlan-share.md
|
||
- clients/dataforth/session-logs/2026-06-01-cbell-m365-bobbi-outlook.md
|
||
- clients/dataforth/session-logs/2026-06-02-session.md
|
||
- clients/dataforth/session-logs/2026-06-04-session.md
|
||
- clients/dataforth/session-logs/project_ad2_context.md
|
||
- clients/dataforth/session-logs/project_pipeline_rebuilt.md
|
||
- clients/dataforth/session-logs/project_test_datasheet_pipeline.md
|
||
- clients/dataforth/session-logs/project_new_product_lines.md
|
||
- clients/dataforth/migration-gap-diff-RESUME.md
|
||
- clients/dataforth/CLAUDE.dataforth.md
|
||
- projects/dataforth-dos/CONTEXT.md
|
||
- session-logs/2026-06-05-session.md
|
||
- session-logs/2026-06/2026-06-09-mike-dataforth-freepbx-safesite-forensics.md
|
||
- session-logs/2026-06/2026-06-18-mike-testdatadb-render-and-security-app.md
|
||
- session-logs/2026-06/2026-06-29-mike-dataforth-nwtoc-pst-deletion-scope-birthbio-corruption.md
|
||
- session-logs/2026-07/2026-07-13-howard-reliant-phones-dataforth-rdp-signing.md
|
||
- .claude/memory/project_dataforth_incident_2026-03-27.md
|
||
- .claude/memory/project_datasheet_pipeline.md
|
||
- .claude/memory/project_neptune_sbr_email_routing.md
|
||
- .claude/memory/reference_dataforth_contact.md
|
||
- .claude/memory/reference_neptune_access_d2testnas.md
|
||
- .claude/memory/feedback_d2testnas_ssh.md
|
||
- .claude/memory/infra_office_network.md
|
||
- .claude/memory/project_dataforth.md
|
||
- .claude/memory/project_dataforth_history.md
|
||
- .claude/memory/project_ad2_dataforth_fork.md
|
||
- .claude/memory/ad2-ssh-mtu-blackhole.md
|
||
- .claude/memory/ad2-comms-via-sync-only.md
|
||
- clients/dataforth/session-logs/2026-06/2026-06-23-mike-pbx-no-inbound-calls-fix.md
|
||
- clients/dataforth/session-logs/2026-06/2026-06-25-howard-dforth-ship-tdr-bsod.md
|
||
- clients/dataforth/session-logs/2026-07/2026-07-01-mike-dataforth-test-data-chain-audit.md
|
||
- clients/dataforth/session-logs/2026-07/2026-07-04-howard-mydata-tpsys-smt-controller-access.md
|
||
- clients/dataforth/docs/audits/2026-07-01-test-data-chain-audit-AD2.md
|
||
- clients/dataforth/session-logs/2026-07/2026-07-05-mike-dos-spec-fix-telemetry-udm.md
|
||
- clients/dataforth/session-logs/2026-07/2026-07-06-mike-sage-sql-peter-session-reset.md
|
||
- clients/dataforth/session-logs/2026-07/2026-07-06-winter-delfina-lopez-new-hire-account.md
|
||
- clients/dataforth/session-logs/2026-07/2026-07-10-mike-haubner-ltobey-address-invalid.md
|
||
- clients/dataforth/session-logs/2026-07/2026-07-14-howard-sage-printing-guru-user.md
|
||
- clients/dataforth/session-logs/2026-07/2026-07-14-howard-pbx-inbound-dnat-flush-fix.md
|
||
- .claude/memory/reference_rmm_spawn_headless_claude.md
|
||
backlinks:
|
||
- projects/dataforth-dos
|
||
- systems/jupiter
|
||
---
|
||
|
||
# Dataforth Corporation
|
||
|
||
Signal conditioning / data acquisition manufacturer in Tucson, AZ. Long-standing ACG client. Active managed relationship — monthly prepaid block. Notable for 64 MS-DOS 6.22 test stations, a major security incident in March 2026, an ongoing test datasheet pipeline modernization project (DOS spec distribution fixed fleet-wide 2026-07-05), an incomplete 2025 post-ransomware recovery restore that silently dropped files across multiple shares (active audit underway), a shares/permissions remediation project (Phase 1 still pending client input), a legacy Fedora Core 3 SMT line controller discovered and recovered in July 2026, and Sage Pro 200 ERP running as an RDS RemoteApp on SAGE-SQL with RDP-redirected printing.
|
||
|
||
---
|
||
|
||
## Profile
|
||
|
||
- **Contract type:** Prepaid hour block (monthly replenishment invoice $2,098.87)
|
||
- **Hours remaining:** 24.75 hrs as of 2026-07-14 (live Syncro `GET /customers/578095` before billing)
|
||
- **Syncro customer ID:** 578095
|
||
- **Syncro managed assets:** 50
|
||
- **Billing rate:** All invoices $0.00 — hours drawn from prepaid block
|
||
- **Invoice CC:** jantar@dataforth.com
|
||
|
||
**Key contacts:**
|
||
|
||
| Name | Username | Role | Email |
|
||
|---|---|---|---|
|
||
| Dan Center | dcenter | Operations (primary IT contact) | dcenter@dataforth.com |
|
||
| John Lehman | jlehman | Engineering, QB code, test specs | jlehman@dataforth.com |
|
||
| Peter Iliya | pIliya | Applications Engineer | pIliya@dataforth.com |
|
||
| Georg Haubner | ghaubner | Engineering; D: drive on HGHAUBNER has pre-ransomware backup of all DF shares | ghaubner@dataforth.com |
|
||
| Kevin Wackerly | kwackerly | IT/Admin, handles calibration@ account | kwackerly@dataforth.com |
|
||
| Logan Tobey | ltobey | Support/Sales | ltobey@dataforth.com |
|
||
| Ben Wadzinski | bwadzinski | Engineering | — |
|
||
| Lee Payne | lpayne | Engineering | — |
|
||
| Theresa Dean | tdean | Admin | tdean@dataforth.com |
|
||
| Joel Lohr | jlohr | **RETIRED 2026-03-31** — account kept enabled; inbox rule forwards ntirety.com notifications to mike@azcomputerguru.com | jlohr@dataforth.com |
|
||
| Ken Hoffman | khoffman / oemdata | TestDataSheetUploader author (external); owns Dataforth product API | — |
|
||
| Delfina Lopez | dlopez | New hire (account created 2026-07-06, OU=Azure_Users) | DLopez@dataforth.com |
|
||
| Winter Williams | — | Dataforth contact for Syncro cleanup requests | — |
|
||
|
||
- **External distributor:** Ginger (gy@quatronix-cn.com) — Quatronix China; receives datasheets
|
||
- **Address:** 3331 E. Hemisphere Loop, Tucson, AZ 85706 | (520) 741-1404
|
||
|
||
**Open Syncro tickets (as of 2026-07-14):**
|
||
|
||
| Ticket | Subject | Status | Date |
|
||
|---|---|---|---|
|
||
| #32505 | Microsoft Access/Log-in request recent hire | Customer Reply | 2026-07-06 |
|
||
| #32489 | DOS test stations not pulling updated spec files (NWTOC v5.0 regression) | Customer Reply | 2026-06-29 |
|
||
| #32441 | Test datasheet generation defects + Phytec recovery | New | 2026-06-17 |
|
||
| #32399 | 10 x Flex Mini Switch installation | New | 2026-06-09 |
|
||
|
||
---
|
||
|
||
## Syncro Asset Reconciliation (2026-06-02)
|
||
|
||
Pulled 78 assets for customer_id `578095`. Syncro managed count 50 (live, 2026-07-14). Fleet-wide Syncro agent break ~2025-10-06 silenced ~half the fleet while machines stayed online in ScreenConnect.
|
||
|
||
| Bucket | Count |
|
||
|---|---|
|
||
| KEEP | 20 |
|
||
| SAVE + FLAG (alive in SC/BD, Syncro agent broken — reinstall) | 21 |
|
||
| REMOVE (dead in all three systems) | 28 |
|
||
| VERIFY (servers, no agent anywhere) | 9 |
|
||
|
||
**Governing rule:** machine is saved if online within 150 days in ANY of Syncro, ScreenConnect, or Bitdefender.
|
||
|
||
**API delete not available** — `DELETE /customer_assets/{id}` returns HTML 404 for current integration token. All deletions via Syncro GUI only.
|
||
|
||
**Coord todo tree** (assigned Howard, parent `103c48ad-7b31-4967-9388-065a91888e7c`): delete 28 dead assets in GUI, decide 9 VERIFY servers, reinstall Syncro agent on 21 SAVE+FLAG machines, switch to metered billing once clean.
|
||
|
||
---
|
||
|
||
## 2026-07-08 GPS-RMM Reconciliation
|
||
|
||
GPS billing = 49 workstation + 3 server = 52 (matches RMM 52) per Winter/Mike.
|
||
|
||
---
|
||
|
||
## Infrastructure
|
||
|
||
### Servers & Services
|
||
|
||
| Host | IP | Role | OS | Notes |
|
||
|---|---|---|---|---|
|
||
| AD1 | 192.168.0.27 | Primary DC, DNS, FSMO roles, Engineering share | Windows Server 2016 | C:\ at **90%** capacity (C:\Engineering = 787 GB) — critical risk. Image plan (`Image2025`) + Files plan (NBF, daily 2 AM, 180-day retention, `C:\Engineering` + `C:\Shares\ITSvc`). GuruRMM agent `bf7bc5ee-4167-4a62-912a-c88b11a5943d`. |
|
||
| AD2 | 192.168.0.6 | Secondary DC, TestDataDB service host, NAS mirror, WebShare | Windows Server 2019 (10.0.17763) | Node.js TestDataDB on :3000. Wiped by 2025 crypto attack — rebuilt. Windows Firewall disabled (all profiles). Shares: `C:\Shares\{c-drive,e-drive,webshare,test}`. Old `D:\c-drive` volume gone — D: = mounted Windows install ISO. MSP360 agent `C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe`; storage account `ACG-Dataforth`. GuruRMM agent **re-enrolled 2026-07-05: new id `4b9fe588-2a86-4855-912e-d90cf5f5e3e4`** (old `cfa93bb6-…` stale). No shadow copies. Runs ClaudeTools on `ad2` branch; coord-API isolated — comms via git sync only. |
|
||
| FILES-D1 | 192.168.0.189 | File server | Windows Server 2016 | Shares: `E:\Shares\{sales,archive}`. `staff` share **MISSING** (see Patterns). GuruRMM agent `8566a19d-49a9-4f8b-9c6c-012cc934484b`. |
|
||
| SAGE-SQL | 192.168.0.153 | Sage ERP (S:), RDS Session Host / Connection Broker / Web Access | Windows Server 2016 | Hosts Sage Pro 200 ERP 2010 (`S:\sbt\sbtw.exe`, Visual FoxPro). ~10 concurrent RDP users. RDS licensing grace period reset 2026-05-06. TSGateway disabled (not externally exposed). Self-signed cert `CN=sage-sql.intranet.dataforth.com`. Bitdefender GravityZone managed AV. Share: `C:\sage`. GuruRMM agent `120ba7bf-8544-48a0-98a1-40ed5cdd3e1f`. **Sage printers = RDP-redirected (Easy Print) only** — native Brother/HP drivers NOT installed server-side (event 1111s); every session default = Microsoft Print to PDF. Sage app credential vault: `clients/dataforth/sage`. |
|
||
| 3CX | 192.168.0.125 | Phone system (possibly inactive) | — | Last logon Oct 2025. Production phones live on VLAN 100 under Sangoma/FreePBX — 3CX role likely superseded. |
|
||
| DF-HYPERV-B | 192.168.0.123 | Hyper-V hypervisor | Windows Server 2025 | GuruRMM enrolled. Newest server in environment. VM inventory not captured. |
|
||
| DF-SVR-D2-Sync | — | (role TBD) | — | GuruRMM enrolled. |
|
||
| ENG-DEV-SERVER | 192.168.0.126 | Engineering dev server | Windows 11 Pro | GuruRMM enrolled. |
|
||
| D2TESTNAS | 192.168.0.9 | SMB1 bridge for DOS test stations + AOI XP backup; rsync source/target for the test-data pipeline | Debian 13 (trixie), Samba 4.22.6 | Repurposed Netgear ReadyNAS. SMB1 enabled globally (CORE..SMB3, NTLMv1) — required for DOS 6.22 stations. rsync daemon port 873 (module `test`, user `rsync`, hosts allow 192.168.0.0/24 + 172.16.0.0/12). SSH: `root@192.168.0.9`. Tailscale: subnet router, **advertises 192.168.100.0/24 ONLY** (not 192.168.0.0/24). Shares: `test`/`datasheets`/`snapshots` (guest; `hosts deny 192.168.1.175`), `aoibackup` (XP-only). Jump host for UDM SSH (direct-tcpip to 192.168.0.254). **2026-07-01 audit:** root SSH key auth from AD2 is broken (publickey denied) — no impact on live rsync-daemon sync, but dormant SCP fallback script would fail. |
|
||
| ESXi hosts | 192.168.0.122, 192.168.0.124 | VMware ESXi hypervisors | ESXi | — |
|
||
| UDM Firewall | 192.168.0.254 | Perimeter firewall/router | UniFi OS 5.1.15 | MAC d0:21:f9:6c:11:02; also responds on 192.168.0.1. SSH: `azcomputerguru@192.168.0.254` (2FA push) or root (keyboard-interactive password, vault `clients/dataforth/udm.sops.yaml`). Boot scripts `/data/on_boot.d/`: `10-neptune-snat.sh`, `20-acg-network-isolation.sh`, `30-freepbx-sip-forward.sh`. **[WARNING] UniFi Protect, Access, and Talk were uninstalled 2026-07-05** (were consuming ~1 GB RAM on a 4 GB device; Dataforth uses FreePBX, none were in use). UniFi Protect 7.1.83, Access 4.2.29, Talk 5.2.7 removed. C2 IPs blocked via iptables (NOT permanent — add to UniFi UI). |
|
||
| PBX (Sangoma FreePBX) | 192.168.100.2 | VoIP PBX — production phones on 192.168.100.0/24 | Sangoma FreePBX 17 / Asterisk 22.5.2, Debian 12 | FirstDigital PJSIP trunk; SBC 66.7.123.215:5060 (Sonus), match 66.7.123.0/24; IP-auth. `qualify_frequency=0` (FD SBC ignores OPTIONS — do NOT revert). TFTP provisioning for Cisco SPA502G phones. Extensions 201–343. SSH: `sangoma@192.168.100.2`. `sudo -S` with password grants root (older "sudo not authorized" note was wrong — corrected 2026-07-14). Vault: `clients/dataforth/pbx.sops.yaml`. [WARNING] Re-apply `PJSip.class.php` line-504 patch after any `fwconsole ma updateall`. **Durable fix applied 2026-07-14:** native UniFi UI Port Forwarding policy "FreePBX-SIP" (WAN1 udp/5060, source-limited 66.7.123.0/24 → 192.168.100.2:5060) — provision-persistent, supersedes the on_boot.d script as primary; script retained as idempotent backup. |
|
||
| **MYDATA TPSys SMT Controller** (`myserver`) | 192.168.1.1 (VLAN 2, SSH-confirmed 2026-07-04) | MYDATA/Mycronic TPSys SMT pick-and-place line controller | Fedora Core 3 "Heidelberg" (Nov 2004), kernel 2.6.16.20, glibc ~2.3.5, bash 3.00, LILO bootloader, SysV init | Discovered + root recovered 2026-07-04. On VLAN 2 "mydata" (192.168.1.0/24). **[WARNING] 192.168.1.1 collides with VLAN 2 gateway address — reconcile whether this box IS the gateway or there is an IP conflict with the UDM.** TPSys UI runs under X (autologin, `.profile` → `startx` on tty1, default runlevel 3 not 5); local PostgreSQL backs TPSys. Root password reset via LILO recovery and vaulted: `clients/dataforth/mydata-smt.sops.yaml`. ACG RSA key: `clients/dataforth/mydata-smt-ssh-key.sops.yaml` (FC3 OpenSSH 3.9 needs legacy `-o KexAlgorithms=+diffie-hellman-group14-sha1 -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa -o MACs=+hmac-sha1 -o Ciphers=+aes128-cbc`). **[CRITICAL] root must NOT be passwordless AND `tpsys` must remain in `wheel` with `pam_wheel.so trust use_uid` uncommented in `/etc/pam.d/su`** — TPSys launcher `/home/tpsys/bin/go` escalates via `su -c` with no tty; a root password without the wheel trust breaks all escalation → X comes up empty (fvwm2 only, no TPSys UI). DR backup (2026-07-04): partition table + MBR/LILO + raw `dd` of OS partitions + `/home` tar + `pg_dumpall`, ~640 MB, SHA256-verified. On-site copy: `D2TESTNAS:/root/dr-backups/mydata-tpsys/2026-07-04/`. Offsite: `AD1 C:\Shares\ITSvc\DR\mydata-tpsys\2026-07-04\` (inside AD1's MSP360 Files plan → B2 `ACG-Dataforth`, nightly 2 AM). **GuruRMM agent cannot run on this box** (glibc ~2.3.5 <2.17, kernel 2.6.16 <2.6.32, SysV init, no systemd). |
|
||
|
||
**Neptune Exchange (ACG infrastructure, physically at Dataforth D2):**
|
||
- `neptune.acghosting.com` | internal `172.16.3.11` | external inbound `67.206.163.124` / outbound `67.206.163.122`
|
||
- Exchange Server 2016, ACG-hosted mail server. Physically colocated at Dataforth D2 — NOT on ACG office LAN despite 172.16.x.x IP.
|
||
- Access requires routing through D2TESTNAS (192.168.0.9).
|
||
- SNAT rule: `/data/on_boot.d/10-neptune-snat.sh` on UDM forces Neptune outbound to `.124`.
|
||
- Vault: `clients/dataforth/neptune-exchange.sops.yaml`
|
||
- [WARNING] TODO: Resubnet Dataforth UDM to a non-overlapping range to permanently fix Neptune routing.
|
||
|
||
### RMM-Spawned Claude on AD2
|
||
|
||
Proven capability (2026-07-01): a headless `claude -p` can be launched on AD2 through its GuruRMM agent (`context:user_session`, detached), writing a deliverable file + `DONE.txt` marker for a background poller. Bypasses AD2's coord-API isolation for read-only investigation. Must `Remove-Item Env:\ANTHROPIC_API_KEY` before invoking — a stale 108-char machine-level key shadows the working OAuth creds at `C:\Users\sysadmin\.claude\.credentials.json`. Reference: `.claude/memory/reference_rmm_spawn_headless_claude.md`.
|
||
|
||
### Share -> Server -> Physical Path Map
|
||
|
||
| Drive/Share | Server | Physical path | Notes |
|
||
|---|---|---|---|
|
||
| Q: / `c-drive` | AD2 | `C:\Shares\c-drive` | Old `D:\c-drive` is gone (D: = mounted install ISO) |
|
||
| T: / `e-drive` | AD2 | `C:\Shares\e-drive` | DOS stations map T: to `\\D2TESTNAS\test` (separate) |
|
||
| X: / `webshare` | AD2 | `C:\Shares\webshare` | DOS stations map X: to `\\D2TESTNAS\datasheets` (separate) |
|
||
| S: / `sage` | SAGE-SQL | `C:\sage` | — |
|
||
| W: / `sales` | FILES-D1 | `E:\Shares\sales` | — |
|
||
| Y: / `archive` | FILES-D1 | `E:\Shares\archive` | — |
|
||
| B: / `Engineering` | AD1 | `C:\Engineering` | — |
|
||
| B: / `itsvc` | AD1 | `C:\Shares\ITSvc` | — |
|
||
| `staff` | FILES-D1 | — | **MISSING** — share does not exist on FILES-D1 |
|
||
|
||
### Workstations (summary)
|
||
|
||
| Category | Count | OS | Notable |
|
||
|---|---|---|---|
|
||
| Engineering | ~12 | Win 10/11 Pro | HGHAUBNER (192.168.0.148) — Georg's PC; D: = full pre-attack backup of all 7 DF shares. GuruRMM agent `2aefe0d5-2357-4bdd-965a-abfccb4767a5`. D1-PWRM for PWRM10 test. |
|
||
| Manufacturing/Assembly | ~14 | Win 10/11 Pro | AS24, AS26 + various assembly/hi-pot stations |
|
||
| Office/Admin | ~12 | Win 10/11 Pro | DF-GAGETRAK (192.168.0.102) — GAGEtrak calibration host. DF-JOEL2 (192.168.0.174) — compromised 2026-03-27, remediated. DESKTOP-U8PHS72 (192.168.0.48) — dlopez/kellynwackerly; local Brother DCP-L2550DW + HP LaserJet. |
|
||
| Shipping | (part of Office/Admin) | Win 10/11 Pro | **DFORTH-Ship** — GuruRMM agent `db17e069-2948-4cbc-97ea-1da721edcaf5`, HP EliteDesk 800 G1 USDT. Recurring BSOD `0x116 VIDEO_TDR_FAILURE` — see Patterns. Do not confuse with near-twin **DForth-Shipp** (`95991b45-d843-4586-8275-9996d0d9ae17`). |
|
||
| End-of-Life (Win 7) | 3 | Windows 7 Pro | LABELPC (192.168.0.100), LABELPC2 (192.168.0.98), D2-RCVG-003 (192.168.0.47) — EOL, on network |
|
||
| AOI Optical Inspection (XP) | 1 | Windows XP | WinXPBE-724667 @ 192.168.1.175 on VLAN 2. Backs up to `\\192.168.0.9\aoibackup` (SMB1, XP-only). EOL. |
|
||
| SMT Line Controller | 1 | Fedora Core 3 "Heidelberg" (2004) | MYDATA TPSys (`myserver`), VLAN 2. Full detail in Servers table. No GuruRMM agent possible. |
|
||
| DOS Test Stations | 64 | MS-DOS 6.22 | TS-1 through TS-30 + variants. Not domain-joined. SMB1 via D2TESTNAS. 14 remain active (confirmed 2026-07-05). |
|
||
|
||
### Email & Identity
|
||
|
||
- **M365 tenant:** dataforth.com | Tenant ID: `7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584`
|
||
- **Entra ID Sync:** Yes — Azure AD Connect on AD1 (ADSync service). Synced OUs: **OU=SyncedUsers** and **OU=Azure_Users**. CompanyUsers OU does NOT sync.
|
||
- **M365 licenses:** 50x Business Premium (40 used as of 2026-07-06), 19x Exchange Online Plan 1 (5 used), 5x SPB (4 used). Note: `O365_BUSINESS_PREMIUM` SKU (f245ecc8) is the legacy part number for M365 Business Standard, distinct from the 5-seat true Business Premium (`SPB`).
|
||
- **SMTP settings:** smtp.office365.com, port 587, STARTTLS — use `sysadmin@dataforth.com`
|
||
- **SMTP AUTH status:** Tenant-level not disabled; per-mailbox varies. `calibration@dataforth.com` SmtpClientAuthentication=true (re-enabled 2026-04-23). `sysadmin@dataforth.com` SMTP AUTH blocked by Exchange Online default — testdatadb uses Graph API for email (Mail.Send permission granted to Claude-Code-M365 app 2026-05-12).
|
||
- **Mail security stack (layered):**
|
||
1. **INKY PhishFence** — transport rule `B859327F-3FBD-4BE7-A47A-97D02F1558A7`, StopProcessingRules=true. Use inbox rules for per-mailbox routing, NOT transport rules.
|
||
2. **Mailprotector CloudFilter** — outbound delivery gateway (`dataforth-com.outbound.emailservice.io`, 52.3.213.180). Active outbound connector "Outbound-Mailprotector". If a message shows "Delivered" in outbound trace but never arrives, check Mailprotector (`/mailprotector skill`).
|
||
- **DKIM:** Both selectors published. Rotated 2026-05-12; cutover to selector2 on 2026-05-16.
|
||
- **DNS Host:** ntirety.com — Dataforth's public DNS zone managed through ntirety's portal. DNS change requests go to ntirety, not a registrar panel. Joel Lohr's account retained to receive ntirety.com notifications (inbox rule → mike@azcomputerguru.com).
|
||
- **AutoForwarding:** blocked by default (tenant outbound spam policy). If per-user forwarding needed, create scoped HostedOutboundSpamFilterPolicy with AutoForwardingMode=On.
|
||
- **MFA:** 3 Conditional Access policies enforced 2026-04-04:
|
||
- "ACG - Require MFA for All Users" — skip from office IP 67.206.163.122
|
||
- "ACG - Block Foreign Sign-Ins" — US-only; MFA-Travel-Bypass group for exceptions
|
||
- "ACG - Block Legacy Authentication"
|
||
- **Named locations:** Dataforth Office - Tucson (67.206.163.122/32, trusted), Allowed Countries - US Only
|
||
- **MFA-Excluded-BreakGlass group:** Brian Faires, Dataforth Calibration, Dataforth Notifications, Endcap, Tablet 01
|
||
- **Get-MessageTrace deprecated Sept 2025:** Use Get-MessageTraceV2 and Get-MessageTraceDetailV2.
|
||
|
||
### Network
|
||
|
||
- **Domain:** intranet.dataforth.com | Forest/Domain Level: Windows Server 2016
|
||
- **ISP:** fdtnet.net | Public IP: 67.206.163.122 (outbound), 67.206.163.124 (Neptune inbound)
|
||
- **Firewall/Router:** UniFi Dream Machine Pro at 192.168.0.254 (also 192.168.0.1), UniFi OS 5.1.15
|
||
- **Network layout:**
|
||
- Main LAN: 192.168.0.0/24 (flat — servers, workstations, DOS stations)
|
||
- Voice/PBX: VLAN 100 (192.168.100.0/24) — production phones. PBX at .2/.196.
|
||
- VLAN 2 "mydata" (192.168.1.0/24): SMT production-line network; gateway 192.168.1.1. Members: WinXPBE-724667 (.175), goldstar19, DESKTOP-FT0T4MK, My9-PC, MYDATA TPSys (`myserver`, .1), + 3 unnamed industrial MACs (00:90:fb:80:f0:c6, 00:80:79:05:23:f2, 00:80:79:04:47:e7). Inter-VLAN routing mydata → main LAN currently OPEN.
|
||
- **VPN:** OpenVPN for ACG remote access. Client subnet 192.168.6.x (GURU-5070 gets 192.168.6.2). [WARNING] GURU-5070 OpenVPN adapter "Local Area Connection" (ifIndex 12) MTU must be 1400 — default 1500 causes PMTU blackhole (tunnel path MTU ~1424; bulk SSH/SCP silently drops). Verify/re-apply: `Set-NetIPInterface -InterfaceIndex 12 -AddressFamily IPv4 -NlMtuBytes 1400`. Permanent fix: `mssfix 1360` server-side.
|
||
- **Drive mappings (GPO):** B: (\\ad1\itsvc), Q: (\\ad2\c-drive), S: (\\SAGE-SQL\sage), T: (\\ad2\e-drive), W: (\\files-d1\sales), X: (\\ad2\webshare), Y: (\\files-d1\archive). DOS stations: T: (\\D2TESTNAS\test), X: (\\D2TESTNAS\datasheets).
|
||
|
||
### GuruRMM Enrollment
|
||
|
||
- **Site name:** Dataforth D1 | Site ID: `3a2f6866-26cd-452c-9806-a8df21475c3c`
|
||
- **Site API key:** vault `clients/dataforth/...` [check vault]
|
||
- **Fleet size:** 45+ agents enrolled; Syncro managed count 50 as of 2026-07-14
|
||
|
||
**[WARNING] GuruRMM enrollment workaround:** WebSocket auth in `ws/mod.rs` does not validate `enrolled_agents.agent_key_hash`. New agent installs must overwrite registry AgentKey with the site API key and restart service. See Gitea issue #8.
|
||
|
||
**[WARNING] `rmm-api.azcomputerguru.com` must be grey-clouded** (DNS-only, not Cloudflare-proxied) — proxy blocks WebSocket. Gitea issue #9.
|
||
|
||
**Known enrolled agents:**
|
||
|
||
| Host | Agent ID | Notes |
|
||
|---|---|---|
|
||
| AD1 | `bf7bc5ee-4167-4a62-912a-c88b11a5943d` | Enrolled 2026-06-04 |
|
||
| AD2 | `4b9fe588-2a86-4855-912e-d90cf5f5e3e4` | Re-enrolled 2026-07-05 (old `cfa93bb6-…` stale) |
|
||
| FILES-D1 | `8566a19d-49a9-4f8b-9c6c-012cc934484b` | Enrolled 2026-06-04 |
|
||
| SAGE-SQL | `120ba7bf-8544-48a0-98a1-40ed5cdd3e1f` | Enrolled 2026-06-04 |
|
||
| DF-HYPERV-B | (see RMM dashboard) | Enrolled 2026-06-04 |
|
||
| DF-SVR-D2-Sync | (see RMM dashboard) | Enrolled 2026-06-04 |
|
||
| ENG-DEV-SERVER | (see RMM dashboard) | Enrolled 2026-06-04 |
|
||
| DF-GAGETRAK | `7626d82c-0736-47a6-8bc6-68e39859caed` | Enrolled 2026-04-23 (auth workaround applied) |
|
||
| HGHAUBNER | `2aefe0d5-2357-4bdd-965a-abfccb4767a5` | Georg's PC; pre-attack backup on D: |
|
||
| DFORTH-Ship | `db17e069-2948-4cbc-97ea-1da721edcaf5` | Recurring TDR BSOD — see Patterns |
|
||
| DESKTOP-U8PHS72 | `25a75932-31cf-4b33-b94b-f4c404830877` | dlopez/kellynwackerly workstation |
|
||
| (additional agents) | — | Full list in GuruRMM dashboard |
|
||
|
||
**Cannot be enrolled:**
|
||
|
||
| Host | Reason |
|
||
|---|---|
|
||
| MYDATA TPSys (`myserver`) | Fedora Core 3: glibc ~2.3.5 (<2.17), kernel 2.6.16 (<2.6.32), SysV init — three hard blockers. Plan agentless monitoring (ICMP/TCP or SSH heartbeat from D2TESTNAS). |
|
||
|
||
### Backup Architecture
|
||
|
||
- **MSP360 ("ACG-Online Backup", `cbb.exe`):** Storage account `ACG-Dataforth` (account ID `0b49ca5e-…`).
|
||
- **AD2:** `AD2 Image` image plan + `Files` plan (180-day retention, NBF, daily 2 AM, `C:\Shares` tree; GFS off, synthetic full, compression, fast-NTFS). No shadow copies on AD2.
|
||
- **AD1:** `Image2025` image plan + Files plan (NBF, daily 2 AM, 180-day retention, covers `C:\Engineering` + `C:\Shares\ITSvc`; created 2026-06-05). Both image and file plans now match AD2.
|
||
- **Pre-attack backup (offline):** HGHAUBNER `D:` drive holds a full pre-attack snapshot of all 7 mapped DF shares. The only recovery source predating the 2025 attack. Access via GuruRMM `user_session` on HGHAUBNER. Cross-machine writes use existing GPO-mapped drives only.
|
||
- **Historical file-level backup:** NBF bunch `faad5a67` in `ACG-Dataforth` storage — restore points 8/29–9/29/2025, archived at old `D:\c-drive\...` path (pre-migration layout).
|
||
- **WizTree backup CSV (2026-06-04):** Full-drive export of HGHAUBNER's D: at AD2 `C:\ClaudeTools\clients\dataforth\WizTree_20260604184904.zip` (sensitive — keep off shares).
|
||
- **MYDATA TPSys DR (2026-07-04):** hybrid image on D2TESTNAS + offsite via AD1 MSP360 Files plan → B2 `ACG-Dataforth`. See Servers table. TODO: full offline `dd`/Clonezilla image in a maintenance window.
|
||
|
||
### Key Applications
|
||
|
||
| Application | Host | URL/Port | Notes |
|
||
|---|---|---|---|
|
||
| TestDataDB | AD2 | http://192.168.0.6:3000 | Node.js + Express, PostgreSQL 18.3. Internal LAN only. UI redesigned 2026-06-18. 475,553 test_records as of 2026-07-01 audit; HTTP API uploader is the live web-delivery path. |
|
||
| Sage Pro 200 ERP 2010 | SAGE-SQL | RDS RemoteApp / S:\sbt\sbtw.exe | Visual FoxPro-based. Users RDP to SAGE-SQL and run Sage there — printers = RDP-redirected (Easy Print). Native drivers not on server; default printer is Microsoft Print to PDF. Credential vault: `clients/dataforth/sage` (username: guru). |
|
||
| GageTrak | DF-GAGETRAK (192.168.0.102) | — | Calibration tracking. Sends email via calibration@dataforth.com (SMTP AUTH re-enabled 2026-04-23). GuruRMM enrolled. |
|
||
| Dataforth Product API | Hoffman's servers | https://www.dataforth.com/api/v1/TestReportDataFiles | OAuth2 client_credentials. Vault: `clients/dataforth/api-oauth.sops.yaml`. Token URL: `https://login.dataforth.com/connect/token`. |
|
||
| QuickBASIC 4.5 ATE | 64 DOS stations | T:\ (\\D2TESTNAS\test) | Automated test equipment. 1,470+ product model specs. 14 stations remain active (2026-07-05). Spec distribution pipeline fixed fleet-wide 2026-07-05 (NWTOC v5.1). |
|
||
| Power Monitor SPA | Georg's dev / TBD | — | Vanilla-JS SPA for Dataforth power meters. Gateway architecture designed for PWM.dataforth.com demo. Parked pending Mike↔Georg conversation. `clients/dataforth/power-monitor-demo/`. |
|
||
| MYDATA TPSys | MYDATA TPSys controller (`myserver`, VLAN 2) | Local X UI / local PostgreSQL | SMT pick-and-place line control. Discovered + documented 2026-07-04. See Servers table. |
|
||
|
||
---
|
||
|
||
## Access
|
||
|
||
### Domain / Server Access
|
||
|
||
- **AD2 SSH:** `ssh sysadmin@192.168.0.6` (port 22) — vault: `clients/dataforth/ad2.sops.yaml` → `credentials.password` — strip backslash escape: `sed 's/\\//g'`. MTU-sensitive: GURU-5070 OpenVPN adapter ifIndex 12 must be MTU 1400.
|
||
- **AD1 SSH:** `ssh sysadmin@192.168.0.27` — vault: `clients/dataforth/ad1.sops.yaml`
|
||
- **D2TESTNAS SSH:** `ssh root@192.168.0.9` — vault: `clients/dataforth/d2testnas.sops.yaml` (password corrected 2026-07-14; was backslash-corrupted). Use `root`, NOT `sysadmin`. Tailscale: `root@100.85.152.90`. SSH key from GURU-5070 authorized; **from BEAST: not authorized**. From AD2, root SSH key auth is broken (publickey denied) — live rsync-daemon path unaffected.
|
||
- **D2TESTNAS `aoibackup` share:** `\\192.168.0.9\aoibackup` — Samba user `admin`, `hosts allow = 192.168.1.175` only. Creds: vault `clients/dataforth/d2testnas.sops.yaml → credentials.smb.aoi-*`.
|
||
- **UDM SSH:** `ssh azcomputerguru@192.168.0.254` (2FA push) or root (keyboard-interactive password; GURU-5070 id_ed25519 key also authorized). Jump via D2TESTNAS: paramiko `direct-tcpip` or ProxyJump. Vault: `clients/dataforth/udm.sops.yaml` (`credentials.ssh_root_password` = keyboard-interactive only — `plink -pw` and paramiko `connect(password=)` refused; use paramiko `auth_interactive`). From BEAST: no route to 192.168.0.x — jump through D2TESTNAS (advertises VLAN 100 only, not 192.168.0.x).
|
||
- **SAGE-SQL SSH:** `ssh sysadmin@192.168.0.153` — SSH key at `C:\ProgramData\ssh\administrators_authorized_keys` on SAGE-SQL. Times out from BEAST (no VPN route); use GuruRMM instead.
|
||
- **PBX SSH:** `sangoma@192.168.100.2` — vault: `clients/dataforth/pbx.sops.yaml`. `sudo -S` with same password grants root (note corrected 2026-07-14 — prior "sudo not authorized" entry was wrong).
|
||
- **WinRM (AD2/AD1):** port 5985 — pywinrm with NTLM, user `INTRANET\sysadmin`
|
||
- **HGHAUBNER:** No SSH. Reached via GuruRMM agent `2aefe0d5`. Cross-machine file writes use existing GPO-mapped drives only.
|
||
- **All server passwords:** vault (individual `clients/dataforth/<host>.sops.yaml` per server)
|
||
|
||
### MYDATA TPSys SMT Controller
|
||
|
||
- Root password: vault `clients/dataforth/mydata-smt.sops.yaml`. ACG RSA key: vault `clients/dataforth/mydata-smt-ssh-key.sops.yaml` (requires legacy SSH options).
|
||
- **Recovery if locked out again:** LILO `boot:` → `linux init=/bin/bash rw` → passwordless root shell (bypasses `sulogin`). Then `mount -o remount,rw /` + `passwd root`. Reboot: `reboot -f` or `echo b > /proc/sysrq-trigger`.
|
||
- **DO NOT blank root, remove tpsys from wheel, or re-comment the `pam_wheel.so trust` line** — any of these kills TPSys on next boot (all su escalations fail, X comes up empty).
|
||
- Accounts: `root`, `tpsys` (app user, in wheel), `tpspool` (spool), `postgres` (uid 500, local DB).
|
||
|
||
### M365 / Entra
|
||
|
||
- **M365 admin:** sysadmin@dataforth.com — vault: `clients/dataforth/m365.sops.yaml`
|
||
- **Tenant ID:** `7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584`
|
||
- **Claude-Code-M365 Entra App:** App ID `7a8c0b2e-57fb-4d79-9b5a-4b88d21b1f29`, secret expires 2027-12-22 — vault: `clients/dataforth/m365.sops.yaml → credentials.entra-app`
|
||
- **MSP remediation app suite:** MSP tenant `ce61461e-81a0-4c84-bb4a-7b354a9a356d`. Old single app `fabb3421`/Claude-MSP-Access DELETED 2026-06-14 — do not use. ComputerGuru tiered apps (Exchange Operator `b43e7342`, etc.) all consented 2026-04-23. Vault: `msp-tools/computerguru-*.sops.yaml`.
|
||
- **Contacts.Read not held** by the investigator app in Dataforth tenant (ErrorAccessDenied, confirmed 2026-07-10). Exchange REST returns 401 (Exchange Admin role not assigned to investigator SP in this tenant).
|
||
|
||
### MSP360 Managed Backup API
|
||
|
||
- **Vault:** `msp-tools/msp360-api.sops.yaml`
|
||
- `cbb.exe` path on AD2: `C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe`
|
||
|
||
### Dataforth Product API (Hoffman)
|
||
|
||
- **Vault:** `clients/dataforth/api-oauth.sops.yaml`
|
||
- Token URL: `https://login.dataforth.com/connect/token` | Grant: `client_credentials` | Client ID: `dataforth.onprem.sync` | Scope: `dataforth.web` | Token TTL: 1 hour
|
||
- Endpoints: `GET /api/v1/TestReportDataFiles/{serial}`, `/bulk`, `/stats`
|
||
|
||
### ESXi / Hypervisors
|
||
|
||
- ESXi-122: 192.168.0.122 — vault: `clients/dataforth/esxi-122.sops.yaml`
|
||
- ESXi-124: 192.168.0.124 — vault: `clients/dataforth/esxi-124.sops.yaml`
|
||
|
||
---
|
||
|
||
## Patterns & Known Issues
|
||
|
||
### RDS / SAGE-SQL / Sage Printing
|
||
|
||
- **Sage Pro 200 ERP 2010 is an RDS RemoteApp only** — users RDP to SAGE-SQL (192.168.0.153, Session Host) and run `S:\sbt\sbtw.exe` there. Sage is NOT installed on workstations.
|
||
- **Printers = RDP-redirected (Easy Print)** — native Brother/HP drivers are NOT installed on SAGE-SQL (event 1111 "driver unknown"). Every session default = Microsoft Print to PDF. If a user sees no printers in Sage, first check for stale disconnected RDP sessions on SAGE-SQL.
|
||
- **Stale session printing fix (dlopez, 2026-07-14):** user had two stale sessions (one active since prior day, one disconnected); fresh reconnects kept re-entering the stale sessions with zero printers. Fix: `logoff <id> /v` for all stale sessions via GuruRMM; new session brought printer redirection up correctly. Note: `logoff` exits nonzero on success — use `query user` output as truth, not exit code.
|
||
- **Registry change for RDP behavior** was applied prior to 2026-07-14 (before that session); fresh logon required to take effect. Watch-item: if Sage printing complaints recur with the Easy Print queues, install native Brother DCP-L2550DW + HP Universal PCL 6 drivers on SAGE-SQL.
|
||
- **RDS licensing:** Grace period reset 2026-05-06 by deleting GracePeriod registry key. Purchase RDS CALs (Per User mode, LicensingType=4).
|
||
- **TSGateway:** Disabled on SAGE-SQL (server not externally exposed). Do NOT re-enable without reason.
|
||
- **SSL cert:** Self-signed, `CN=sage-sql.intranet.dataforth.com`. Non-domain machines must import to Trusted Root + Trusted Publishers manually.
|
||
- **Bitdefender GravityZone** managed AV on SAGE-SQL — may block PowerShell execution; temporary disable may be needed for admin work.
|
||
- **Sage credential vault:** `clients/dataforth/sage` (username: guru). Password reset by Dataforth staff 2026-07-14; verified working. Never inline the password.
|
||
|
||
### Active Directory
|
||
|
||
- **No custom security groups** — only default Windows groups. Service accounts in OU=ServiceAccounts.
|
||
- **ClaudeTools-ReadOnly AD account** — purpose unclear. Investigate.
|
||
- **Ken Hoffman has two accounts** (khoffman + oemdata) — not consolidated.
|
||
- **jlohr account retained** — post-retirement (2026-03-31); kept enabled for ntirety.com notifications. Inbox rule forwards to mike@azcomputerguru.com. Do NOT disable.
|
||
- **Entra sync scope:** OU=SyncedUsers **and OU=Azure_Users** sync to Entra. CompanyUsers OU does NOT sync. 38 stale TS-* test station accounts cleaned from Entra 2026-03-27.
|
||
- **New hire pattern (dlopez, 2026-07-06):** create on-prem in OU=Azure_Users, add to RDWeb-Users, trigger ADSync delta. ADSync service runs on AD1 itself.
|
||
- **SKU disambiguation:** `O365_BUSINESS_PREMIUM` (f245ecc8) = M365 Business Standard (legacy part number), NOT true Business Premium (`SPB`). This is what standard user accounts hold.
|
||
|
||
### Voice / Phones / FreePBX
|
||
|
||
- **Production phones VLAN:** 192.168.100.0/24. PBX at .2/.196. All production phones live here.
|
||
- **VLAN 2 (192.168.1.0/24) is the mydata/SMT VLAN** — do NOT confuse with voice. Phones landing on 192.168.1.x cannot reach PBX.
|
||
- **D1-Server-Room port 1:** Controls lobby drop → must stay on VLAN 100. Reverted to default once before (2026-05-04 incident).
|
||
- **FirstDigital trunk — `qualify_frequency=0`:** FD's Sonus SBC ignores SIP OPTIONS keepalives. Non-zero qualify causes trunk to go Unavailable and blocks all INVITEs. **Do NOT revert.**
|
||
- **PJSip.class.php line 504 patch must be re-applied** after any `fwconsole ma updateall` — wiped by FreePBX updates. Backup before each update.
|
||
- **Do NOT port-forward the RTP range (10000-20000)** on the UDM. Static RTP DNAT causes a conntrack collision with the PBX's outbound RTP — inbound works but outbound audio dies. SIP 5060 forward only, source-locked to 66.7.123.0/24.
|
||
- **Inbound SIP DNAT (recurring failure, now durably fixed):** Previously relied only on `/data/on_boot.d/30-freepbx-sip-forward.sh`, which is flushed by any UniFi controller provision mid-uptime (not just reboots). Outage 2026-07-13 11:05 AM → 2026-07-14 09:39 AM fixed by: (1) re-running the on_boot.d script (immediate fix), (2) creating a native UniFi UI Port Forwarding policy "FreePBX-SIP" (WAN1 udp/5060, source 66.7.123.0/24 → 192.168.100.2:5060) that survives provisions. Coord to-do `45572ee1` (durable fix) is now closed. on_boot.d script retained as an idempotent backup. Diagnostic: `iptables -t nat -S | grep -iE '5060|192.168.100.2'` on UDM — empty means rule is gone.
|
||
- **`sangoma` sudo clarification:** `sangoma` is in the `sudo` group and `sudo -S` with the password DOES grant root (prior "sudo not authorized" note was wrong — corrected 2026-07-14). `asterisk -rx` still denied for sangoma (socket is owner-writable only).
|
||
- Prior outages: 2026-06-08 (#32392), 2026-06-23 (#32450), 2026-07-14 (#32553).
|
||
|
||
### Exchange Online / Email
|
||
|
||
- **INKY PhishFence StopProcessingRules:** Kills all subsequent transport rules. Use inbox rules for per-mailbox forwarding, NOT transport rules.
|
||
- **Mailprotector CloudFilter:** Outbound delivery goes through Mailprotector. If "Delivered" per outbound trace but message never arrives, check Mailprotector (`/mailprotector skill`).
|
||
- **AutoForwarding blocked by default** — create scoped HostedOutboundSpamFilterPolicy with AutoForwardingMode=On if needed per sender.
|
||
- **Autocomplete stale-cache trap (Haubner→Tobey, 2026-07-10):** Outlook displays the correct SMTP address from cache but routes via a stored EX/LegacyExchangeDN that no longer resolves. Fix: user deletes the stale autocomplete entry in Outlook and re-picks from GAL. Remote cache-clear more disruptive (file locked while Outlook is open, nukes all cached names).
|
||
- **Contacts.Read / Exchange REST gaps:** The investigator app does not have Contacts.Read in Dataforth (ErrorAccessDenied). Exchange REST returns 401 (Exchange Admin role not assigned to investigator SP in this tenant). Use GuruRMM for workstation-level inspection when these APIs are blocked.
|
||
|
||
### GuruRMM Agent Deployment
|
||
|
||
- **WebSocket auth bug (Issue #8):** `enrolled_agents.agent_key_hash` not validated. Workaround: overwrite registry `HKLM:\SOFTWARE\GuruRMM\AgentKey` with site API key, restart service.
|
||
- **rmm-api.azcomputerguru.com must be grey-clouded** — Cloudflare proxy blocks WebSocket. Gitea Issue #9.
|
||
- **AD2 re-enrolled 2026-07-05:** new agent id `4b9fe588-2a86-4855-912e-d90cf5f5e3e4`. Old id `cfa93bb6-…` is stale — do not dispatch to it.
|
||
- **No RMM agent possible on FC3-class legacy appliances:** confirmed on MYDATA TPSys (glibc ~2.3.5, kernel 2.6.16, SysV init — three hard blockers). Plan agentless monitoring (ICMP/TCP probe or SSH heartbeat from D2TESTNAS).
|
||
- **SYSTEM token cross-machine writes:** GuruRMM `user_session` commands use an impersonated token with no network credentials. Fresh UNC paths fail with Access Denied. Use existing GPO-mapped drives (Q:, T:, etc.) for cross-machine writes — proven on HGHAUBNER 2026-06-04.
|
||
- **Legacy Red-Hat-family recovery pattern:** if locked out, LILO `boot:` → `linux init=/bin/bash rw` → passwordless root shell (bypasses sulogin on Red-Hat family).
|
||
- **RMM dispatch JSON quoting:** inline JSON with `\\` escapes fails the API's JSON parse. Use `ps-encoded.sh` for PowerShell dispatches. `query session` exits nonzero on success — use stdout content as truth.
|
||
|
||
### AD2 SSH / VPN MTU
|
||
|
||
- **PMTU blackhole on GURU-5070 → AD2 SSH:** OpenVPN adapter ifIndex 12 defaults MTU 1500; tunnel path MTU is ~1424. Bulk TCP segments silently dropped; interactive commands pass. Presents as "flaky VPN" or "random lockouts."
|
||
- **Fix (applied 2026-06-18, registry-persistent):** `Set-NetIPInterface -InterfaceIndex 12 -AddressFamily IPv4 -NlMtuBytes 1400` via GURU-5070 RMM agent. May reset on OpenVPN reconnect — re-apply if stalling resumes.
|
||
- **Durable fix:** `mssfix 1360` server-side on the Dataforth OpenVPN server (fleet-wide).
|
||
- **AD2 is NOT the target for SSH diagnosis** when SSH is the failing channel — use RMM.
|
||
|
||
### AD2 Branch / Coordination
|
||
|
||
- **AD2 operates on the `ad2` git branch.** Fork = main + thin Dataforth-specific commits. Never edit shared fleet files on `ad2`. Dataforth context: `clients/dataforth/CLAUDE.dataforth.md`.
|
||
- **AD2 is coord-API isolated:** 172.16.3.30 unreachable from Dataforth LAN. All inter-session coordination via git sync: committed handoff docs + `## Note for <user>` blocks. Do NOT use the coord skill for AD2.
|
||
- **sync.sh push gotcha:** always fails at Phase 4 on `ad2` branch (pushes `main`). Force-push manually: `git push --force-with-lease origin ad2`.
|
||
- **Stale machine-level ANTHROPIC_API_KEY on AD2** (108 chars, invalid) shadows working OAuth creds. Must `Remove-Item Env:\ANTHROPIC_API_KEY` before invoking `claude -p`.
|
||
- **AD2 RMM side channel:** GuruRMM agent (`4b9fe588-…`, current) can spawn a headless `claude -p` for live read-only ground-truth investigation (proven 2026-07-01). Unset the stale API key first.
|
||
|
||
### Post-Ransomware Recovery Restore (2025) — Incomplete File Migration
|
||
|
||
- **The 10/1/2025 recovery restore was incomplete.** `Restore plan 10/1/2025` (~3.4M files) migrated each share from old `D:\<share>` to `C:\Shares\...` on AD2 and dropped files in the process. Proven case: SP1366 MAQ20 — each `PRINTOUTS FOR MANUFACTURING` folder for revisions E–H received only one file when the backup contained ~6 per revision.
|
||
- **Scope unknown.** A full migration-gap audit is underway (WizTree both sides). Review-only — no automatic restore (some deletions were intentional; HGHAUBNER backup is additive-only and includes personal files).
|
||
- **AD2 D: drive is gone.** Old `D:\c-drive` data volume repurposed as mounted Windows install ISO. Historical file-level backup (bunch `faad5a67`) archived data under `D:\c-drive\...` (pre-migration path) — reconcile paths accordingly.
|
||
|
||
### Shares ACL State — All Open to All Staff
|
||
|
||
- **All 8 business shares grant access to every employee** (Everyone/Domain Users Full or Modify on all). No department-based security groups. Payroll, OSHA, Purchase Orders, Accounting/QuickBooks, Sage financials fully readable/writable by all domain users.
|
||
- **Remediation project in progress** (Shares & Permissions, started 2026-06-10). Phase 0 (discovery) complete. Phase 1 (client input) pending discovery email to Dan Center. Do not apply ACL changes until after client sign-off.
|
||
- **Special shares excluded from department-model remediation:** `test` (DOS/SMB1 guest — leave open), `webshare` (preserve `svc_testdatadb:Full`), `ITSvc` (Domain Computers needs Read), Sage app data path.
|
||
- **Phase 2 target-state strawman (drafted 2026-06-22):** `Company\Departments\` tree with `SG-<Resource>-<RW|RO>` groups, `Restricted\` branch with broken inheritance, ABE on. Option A (keep current drive letters, reorganize folders within) recommended for least disruption. Full detail: `clients/dataforth/docs/projects/shares-permissions/`.
|
||
- **Six items gate Phase 2 sign-off:** confirmed department list; department → share access matrix; named access for sensitive data; department rosters; legacy cleanup approval; Engineering destination volume (AD1 C: ~90% full blocks any ENGR restructure).
|
||
|
||
### DOS Test Station Data Pipeline
|
||
|
||
- **DOS spec distribution fixed fleet-wide 2026-07-05** (Syncro #32489 root-caused 2026-07-01, fix deployed 2026-07-05). All 14 active stations confirmed carrying current `5BMAIN.DAT` (83,200, 06-26-26).
|
||
- **Root cause (F1, resolved):** deployed `NWTOC.BAT` v5.0 copied only `*.BAT` and `*.EXE` — zero `.DAT` files — by design (its own changelog said so). No version of NWTOC ever distributed shared `COMMON` master specs; v5.1 added that copy.
|
||
- **F2 (resolved):** `COPY /Y` is not a valid MS-DOS 6.22 switch. Mike confirmed all stations are genuine 6.22 — `/Y` caused NWTOC to copy nothing fleet-wide since 2026-03-16. Fixed in v5.1 (plain `COPY`).
|
||
- **NWTOC v5.1 deployed 2026-07-05** (plain `COPY`, adds one-way `.DAT` pull from `COMMON/ProdSW` into `C:\ATE\<type>DATA`, `ATTRIB -R` guard, CRLF-forced). Validated on TS-3L, promoted fleet-wide. v5.0 backed up to `NWTOC50.BAK`. `CTONWTXT` updated to v2.4 (drops `/Y`). `CTONW` at v5.2 (flat-file telemetry, no network MD) then v5.3 pending (CTONW v5.3 dispatch went to stale AD2 id and needs redoing).
|
||
- **Tier-A telemetry deployed:** `SYSTAT.BAT` (plain paused status page) and flat-file telemetry to `T:\STATUS\%MACHINE%.{CFG,AUT,SPV,MEM}` — 14 stations collected cleanly 2026-07-05.
|
||
- **Rollout is per-station opt-in** (broken v5.0 cannot bootstrap its own replacement — each station needs one manual `COPY` seed, then self-maintains).
|
||
- **Secondary cleanup items (not yet done):** remove stray `TS-21\ProdSW` file (F3 — breaks AD2↔NAS rsync push every run); update `testdatadb\specdata\` from live engineering masters (F4 — frozen 2026-03-27 snapshot); rotate/vault plaintext creds in scripts (F5); retire dead `For_Web` output and abandoned v1.x script drafts.
|
||
- Full 2026-07-01 audit report: `clients/dataforth/docs/audits/2026-07-01-test-data-chain-audit-AD2.md`.
|
||
|
||
### Hardware / Endpoint — Aging Fleet
|
||
|
||
- **DFORTH-Ship recurring TDR BSOD (`0x116 VIDEO_TDR_FAILURE`, diagnosed 2026-06-25):** integrated Intel HD Graphics 4600, driver 20.19.15.5126 (Intel's final, 2020-01-20), on an 11.5-year-old HP EliteDesk 800 G1 USDT (BIOS 2014-12-10). Five minidumps 2025-11-03 through 2026-06-24, accelerating cadence. Mitigation: Edge hardware acceleration disabled via machine policy (`HKLM\SOFTWARE\Policies\Microsoft\Edge\HardwareAccelerationModeEnabled = 0`). No durable fix for integrated graphics — **PC replacement is the real fix**; thermal cleaning is a secondary measure. Verify Edge policy at `edge://policy` after restart. Do not confuse with near-twin **DForth-Shipp** (`95991b45-…`) — verify agent ID before acting.
|
||
- **MYDATA TPSys controller (~20 years old, FC3 Nov 2004):** documented and DR-backed 2026-07-04 after being completely undocumented until discovered at the physical console. Worth a broader sweep for other undocumented devices on VLAN 2 — 3 unnamed industrial MACs already known but unresolved as of 2026-06-01.
|
||
- **End-of-Life machines on network:** 3x Windows 7 (LABELPC, LABELPC2, D2-RCVG-003), 1x Windows XP (WinXPBE-724667/AOI). All EOL, all unpatched.
|
||
|
||
### Security
|
||
|
||
- **C2 IP blocks are iptables only** — do not survive UDM reboot. Must add 80.76.49.18 and 45.88.91.99 (AS399486 Virtuo, Montreal) to permanent UniFi UI block list.
|
||
- **AD1 disk 90% full** — C:\Engineering = 787 GB of 1023 GB. Risk of replication failures. Blocks Engineering share restructure.
|
||
- **Windows Firewall disabled on AD2** (all profiles) — known risk, not remediated.
|
||
- **AD1/AD2 on Server 2016/2019** — approaching end of mainstream support.
|
||
- **Entra ID P2 not licensed** — IdentityRiskyUser risk check returns 403 (no P2).
|
||
- **Plaintext credentials in Dataforth test-data-chain scripts (F5, 2026-07-01):** rsync daemon password in `Sync-FromNAS-rsync.ps1`, Postgres `testdatadb_app` password in `testdatadb\database\db.js` — world-readable via the `test` SMB share. Rotation + vaulting recommended, not yet done.
|
||
- **UniFi Protect/Access/Talk removed 2026-07-05** — were consuming ~1 GB RAM on 4 GB UDM while unused (Dataforth uses FreePBX). All three uninstalled. UDM management plane briefly down during post-uninstall reboot; data plane (forwarding) remained live.
|
||
|
||
### Syncro Asset Management
|
||
|
||
- **Fleet-wide Syncro agent break ~2025-10-06:** ~half of Dataforth machines stopped reporting to Syncro while remaining online in ScreenConnect. Hard cutoff, not gradual attrition. Do NOT auto-remove machines frozen at that date without cross-checking ScreenConnect.
|
||
- **Bitdefender is NOT a liveness signal:** Dataforth phasing off BD; 53 of 57 GravityZone endpoints in "Deleted" folder. Missing from BD = BD agent uninstalled.
|
||
- **API delete not available** — all asset deletions via Syncro GUI (`/customer_assets?customer_id=578095`).
|
||
|
||
### `staff` Share Missing
|
||
|
||
The `staff` network share is absent from FILES-D1 (only `archive` and `sales` exist). HGHAUBNER's backup includes a `DF Staff` folder, suggesting the share existed pre-attack. Not in scope for the current migration-gap diff — separate issue requiring investigation.
|
||
|
||
---
|
||
|
||
## Active Work
|
||
|
||
As of 2026-07-14 (4 open Syncro tickets per live pull):
|
||
|
||
- **DOS Test Station Data Pipeline (Syncro #32489, active):** Root cause confirmed and fix deployed fleet-wide 2026-07-05 (NWTOC v5.1). All 14 active stations confirmed with current specs. Ticket status is Customer Reply — awaiting Dataforth to confirm on their end. Pending: (1) promote `CTONW v5.3` (dispatch to stale AD2 agent id needs redoing with new id `4b9fe588-…`); (2) secondary cleanup items F3–F5 from the 2026-07-01 audit (stray TS-21 file, frozen specdata, plaintext creds in scripts); (3) close ticket once client confirms resolution.
|
||
|
||
- **Test Datasheet Pipeline (Syncro #32441, open):**
|
||
- Production outbound pipeline healthy — 475K+ records, DSCA33/45 recovery complete (1,452 certs published 2026-06-18 via Hoffman API).
|
||
- Email notifications deployed (Graph API via `sysadmin@dataforth.com`).
|
||
- 8B/5B/SCM render gap (~9,624 records unpublished) — 136 templates mined, wiring handed to AD2 (parked).
|
||
- `testdatadb\specdata\` frozen at 2026-03-27 snapshot — any spec limit changed after that date not reflected in server-side generated datasheets even though the outbound pipeline is healthy (F4).
|
||
- 2 niche DSCA models (DSCA33-1948, DSCA45-1746) have no Hoffman original — cannot auto-publish.
|
||
|
||
- **Flex Mini Switch installation (Syncro #32399, open):** 10 x Flex Mini Switch installation. No further detail captured in reviewed session logs.
|
||
|
||
- **New hire account/access (Syncro #32505, Customer Reply):** Delfina Lopez account created 2026-07-06 in AD + M365. Ticket is in Customer Reply — follow up with Winter.
|
||
|
||
- **DFORTH-Ship BSOD (ongoing monitoring):** Edge hardware-acceleration mitigation applied 2026-06-25; needs on-site Edge restart/reboot to take effect. Monitor for recurrence. If it bugchecks again, pull older dump signatures to confirm hardware fault trajectory. Schedule thermal cleaning. Recommend PC replacement as durable fix.
|
||
|
||
- **UDM inbound SIP DNAT (closed 2026-07-14):** Durable fix applied — native UI Port Forwarding rule "FreePBX-SIP" created. Coord to-do `45572ee1` closed. Syncro ticket #32553 (0.5 hr, block debited 25.25 → 24.75 hrs). Monitor stability.
|
||
|
||
- **Shares & Permissions project (Phase 1 — BLOCKING, pending client input):** Phase 0 complete 2026-06-10. Discovery email to Dan Center drafted (`docs/projects/shares-permissions/discovery-email-draft.md`) — **not yet sent** (recipients/sender not locked). Phase 1 blocked on client responses: department list, access matrix, sensitive-data rules, rosters. Phase 2 strawman drafted 2026-06-22. Next step: polish client HTML and send discovery email.
|
||
|
||
- **MYDATA TPSys SMT controller:** Root recovered and documented 2026-07-04. **Outstanding:** (1) reconcile 192.168.1.1 IP vs VLAN 2 gateway conflict; (2) verify tpsys wheel-group + NOPASSWD sudoers change survived the reboot (`id tpsys`, `sudo -l` as tpsys); (3) stand up agentless monitoring (ICMP/TCP probe or SSH heartbeat from D2TESTNAS — inter-VLAN routing to mydata is open); (4) full offline `dd`/Clonezilla image in a maintenance window.
|
||
|
||
- **Migration-gap audit (parked):** WizTree CSV of HGHAUBNER's pre-attack backup captured. WizTree on live servers deferred — no diff yet. Plan: run WizTree on AD2, FILES-D1, SAGE-SQL, AD1 → diff per share → `clients/dataforth/migration-gap-catalog-2026-06-04.md`. Full plan: `clients/dataforth/migration-gap-diff-RESUME.md`.
|
||
|
||
- **Syncro asset cleanup (with Howard):** 28 confirmed-dead assets pending GUI deletion; 21 machines need Syncro agent reinstall; 9 servers in VERIFY bucket. Coord todo tree `103c48ad-…` assigned to Howard. Switch to metered billing once clean.
|
||
|
||
- **AOI XP backup + isolation (ongoing):** AOI XP @ 192.168.1.175 on VLAN 2; `aoibackup` SMB1 share locked to XP only. Optional EOL hardening pending: block XP → company LAN (except NAS 192.168.0.9) + Internet on UDM. Todo `37543f7f`.
|
||
|
||
- **AD2 Claude capability updates (parked):** AD2 needs syncro + coord commands, DF wiki read-write, Dataforth client data access. Coord API unreachable from Dataforth LAN — comms via git sync + RMM-spawn side channel.
|
||
|
||
- **Power Monitor SPA demo (parked):** Georg Haubner's vanilla-JS power-meter SPA. Gateway spec at `clients/dataforth/power-monitor-demo/GATEWAY-SPEC.md`. Parked pending Mike↔Georg conversation.
|
||
|
||
- **MFA enrollment:** 19 users not enrolled at April 4 enforcement date; current enrollment count unverified.
|
||
|
||
- **C2 IP blocks need permanence:** iptables rules on UDM (80.76.49.18, 45.88.91.99) need to be added to permanent UniFi UI block list.
|
||
|
||
- **jlohr forwarding:** ntirety.com inbox rule active as of 2026-05-12. Defunct transport rule pending cleanup.
|
||
|
||
- **RDS / SAGE-SQL:** GPO cert distribution pending. RDS CALs purchase needed long-term.
|
||
|
||
- **Haubner → Tobey autocomplete fix (2026-07-10):** Diagnosed as stale cached EX/LegacyExchangeDN in Outlook autocomplete. Recommended fix: Haubner deletes the Logan entry in Outlook and re-picks from GAL. Awaiting Mike's direction on delivery method (instructions to Haubner vs scheduled remote clear).
|
||
|
||
---
|
||
|
||
## History Highlights
|
||
|
||
| Date | Event |
|
||
|---|---|
|
||
| 2025 | Crypto/ransomware attack — AD2 wiped and rebuilt, many files lost. Test datasheet pipeline broken. |
|
||
| 2025-08-29 – 2025-09-29 | MSP360 file-level backup (`faad5a67`) covering DF shares at old `D:\c-drive\...` path. Last snapshot before the recovery restore. |
|
||
| 2025-10-01 – 2025-10-02 | Post-ransomware recovery restore (`Restore plan 10/1/2025`, ~3.4M files) migrated shares from `D:\<share>` to `C:\Shares\...` on AD2. Restore was incomplete — files dropped in multiple folders. |
|
||
| ~2025-10-06 | Fleet-wide Syncro agent break — ~half of Dataforth machines freeze in Syncro while remaining online in ScreenConnect. Root cause unknown. |
|
||
| 2026-01-19 | DOS Update System built and deployed — NWTOC/CTONW/UPDATE/DEPLOY BAT files, 39 deployments. |
|
||
| 2026-03-20 | Galactic Advisors security assessment — AD1 C: at 90%, legacy SQL 2008 R2 client noted. |
|
||
| 2026-03-27 | **Major security incident:** DF-JOEL2 compromised via social engineering/ScreenConnect (attacker "Angel Raya", C2 on Virtuo hosting). M365 sign-in from Turkey. Full remediation. 3 CA policies deployed. MFA notice sent. IC3 filed (`1c32ade367084be9acd548f23705736f`). |
|
||
| 2026-03-27–29 | Test datasheet pipeline rebuilt — new Node.js pipeline replaces VB6 DFWDS + VB.NET uploader. 72/73 Quatronix datasheets generated. |
|
||
| 2026-03-31 | Joel Lohr retirement. Brian Faires mailbox converted to shared. 38 stale Entra TS-* accounts deleted. |
|
||
| 2026-04-04 | MFA CA policies enforced (switched from report-only). |
|
||
| 2026-04-11–12 | SCMVAS/SCMHVAS pipeline extension — 27,503 records backfilled, 434 Engineering-Tested .txt files imported. |
|
||
| 2026-04-12 | TestDataDB PostgreSQL migration verified (2.89M records). Hoffman API discovered. |
|
||
| 2026-04-13 | API architecture discussion with Hoffman — client_credentials grant confirmed. |
|
||
| 2026-04-14 | DFWDS logic ported to Node.js. 897 staged datasheets drained. 803 new records pushed to Hoffman API. |
|
||
| 2026-04-15 | Major release — DB dedup (2.89M→469K rows), FAIL→PASS retest rule, 170,984 records bulk-pushed to Hoffman. Dashboard upgrades. |
|
||
| 2026-04-23 | Full Dataforth tenant onboarded to all 5 ComputerGuru tiered apps. calibration@ SMTP AUTH fixed. DF-GAGETRAK GuruRMM enrolled. Syncro #32142 billed. |
|
||
| 2026-05-03 | jantar@dataforth.com darkweb breach check — no indicators of compromise. eM Client OAuth grant revoked. |
|
||
| 2026-05-04 | Howard onsite — lobby phone offline (VLAN misconfiguration on D1-Server-Room port 1 → fixed to VLAN 100). |
|
||
| 2026-05-06 | SAGE-SQL RDS issues resolved — grace period reset, SSL cert replaced, TSGateway disabled. |
|
||
| 2026-05-12 | Pipeline audit + email notifications implemented (Graph API). jlohr forwarding configured. DKIM keys rotated. |
|
||
| 2026-06-01 | AOI optical-inspection XP PC isolated onto VLAN 2 @ 192.168.1.175; `aoibackup` SMB1 share created on D2TESTNAS locked to the XP only. D2TESTNAS confirmed Debian 13 / Samba 4.22.6. |
|
||
| 2026-06-01 | Chauncey Bell (cbell) M365 verified; Bobbi's Outlook printing fixed. Ticket #32364 (0.5 hr onsite). |
|
||
| 2026-06-02 | Syncro asset reconciliation (78 assets): 20 keep / 21 save+flag / 28 remove / 9 verify. |
|
||
| 2026-06-04 | SP1366 MAQ20 manufacturing print recovery — 19/20 PDFs for revisions E–H restored from HGHAUBNER pre-attack backup. GuruRMM fleet grew 13 → 45 agents. WizTree backup-side CSV captured. |
|
||
| 2026-06-05 | AD1 Files backup plan created (cbb.exe, NBF, 180-day retention, `C:\Engineering` + `C:\Shares\ITSvc`). |
|
||
| 2026-06-05 | Mailprotector CloudFilter discovered as Dataforth's outbound delivery layer. Email from Georg held by Mailprotector; released manually. `/mailprotector` skill built. |
|
||
| 2026-06-05 | Georg Haubner's Power Monitor SPA analyzed. Gateway architecture designed for PWM.dataforth.com demo. Parked. |
|
||
| 2026-06-08–09 | **Total Dataforth phone outage.** Outbound failed (`qualify_frequency` → trunk Unavailable); inbound never worked (no SIP port-forward existed). Fixed: `qualify_frequency=0` in pjsip DB; PJSip.class.php line 504 re-patched; `30-freepbx-sip-forward.sh` added. Two-way audio verified. Syncro #32392, 1.0 hr emergency (×1.5 rate). |
|
||
| 2026-06-10 | Shares & Permissions Phase 0 complete. All 8 business shares open to all employees. Phase 1 pending. |
|
||
| 2026-06-17 | AD2 identity.json + Python 3.12.8 installed. `CLAUDE.dataforth.md` created. |
|
||
| 2026-06-18 | DSCA33/45 certs recovered via Hoffman API — 1,452 new certs published. `parseRawData` bug root-caused. AD2 SSH PMTU blackhole diagnosed and fixed (MTU 1400). TestDataDB UI redesigned. Syncro #32441. |
|
||
| 2026-06-22 | Shares & Permissions Phase 2 target-state strawman drafted — `Company\Departments\…Restricted\…` tree with `SG-<Resource>-<RW|RO>` groups. Internal draft + client-facing HTML render. |
|
||
| 2026-06-23 | **PBX no-inbound-calls emergency fixed.** UDM SIP 5060 DNAT flushed by UniFi controller provision (not a reboot). Re-ran on_boot.d script; verified inbound restored. Vault `pbx.sops.yaml` password corrected. Syncro #32450, 1.5 hr emergency (×1.5 rate), block 31.5 → 30.0 hrs. |
|
||
| 2026-06-24–25 | DFORTH-Ship recurring BSOD diagnosed (0x116, Intel HD 4600, 11.5-yr-old EliteDesk). Edge hardware-acceleration mitigation applied. PC replacement recommended. |
|
||
| 2026-07-01 | Test-data-chain ground-truth audit via RMM-spawned headless Claude on AD2. Confirmed NWTOC v5.0 never copies .DAT files (F1); `COPY /Y` invalid on 6.22 (F2); three additional findings. Stale assumptions corrected (PostgreSQL 18, rsync-daemon sync path, HTTP API uploader is live web-delivery). |
|
||
| 2026-07-04 | MYDATA TPSys SMT controller (myserver, FC3/VLAN2) discovered + root recovered via LILO single-user; vaulted; RMM agent ruled out. DR backup taken and stored on-site + offsite. |
|
||
| 2026-07-05 | **DOS spec distribution fixed fleet-wide.** NWTOC v5.1 deployed (adds .DAT pull, plain COPY, no /Y). All 14 active DOS stations confirmed with current specs. Tier-A station telemetry added. UDM: UniFi Protect/Access/Talk uninstalled (memory savings). `unifi.service` restarted (0-clients issue — RabbitMQ/WebSocket memory starvation). |
|
||
| 2026-07-06 | Delfina Lopez (dlopez) new hire account created in AD (OU=Azure_Users) + M365 (Business Standard). Ticket #32505. |
|
||
| 2026-07-06 | SAGE-SQL session reset for piliya (Peter Iliya, disconnected session 13). Ticket #32506, no billing. |
|
||
| 2026-07-08 | GPS-RMM billing reconciliation: 49 workstation + 3 server = 52 (matches RMM). |
|
||
| 2026-07-10 | Haubner→Tobey "address invalid" diagnosed — stale autocomplete EX/LegacyExchangeDN on Haubner's Outlook. Awaiting fix delivery decision. |
|
||
| 2026-07-13 – 2026-07-14 | **PBX inbound outage** (2nd UDM SIP DNAT flush, 07-13 ~11:05 AM → 07-14 09:39 AM AZ). Fixed by re-running on_boot.d script; **durable fix:** native UI Port Forwarding policy "FreePBX-SIP" created (provision-persistent). Coord to-do `45572ee1` closed. Vault: UDM root password corrected (keyboard-interactive only); d2testnas password de-corrupted. Sangoma sudo-works note corrected. Syncro #32553, 0.5 hr, block 25.25 → 24.75 hrs. |
|
||
| 2026-07-14 | **Sage printing fix (dlopez).** Diagnosed: stale RDP sessions on SAGE-SQL (no redirected printers until logged off). Logged off stale sessions via GuruRMM; fresh RDP brought printer redirection up. Sage app credential (guru user) vaulted at `clients/dataforth/sage`. Syncro #32552, 0.5 hr, block 25.75 → 25.25 hrs (then 25.25 → 24.75 from #32553 same day; final live block = 24.75). |
|
||
|
||
---
|
||
|
||
## Backlinks
|
||
|
||
- [[projects/dataforth-dos]] — Active test datasheet pipeline project on AD2
|