diff --git a/.claude/memory/MEMORY.md b/.claude/memory/MEMORY.md index caa1b1eb..f14961dc 100644 --- a/.claude/memory/MEMORY.md +++ b/.claude/memory/MEMORY.md @@ -210,3 +210,4 @@ - [RMM deploy via ScreenConnect](reference_rmm_deploy_via_screenconnect.md) — push GuruRMM agent to client workstations via SC send-command (SYSTEM), not DC remote-exec (DCOM/schtasks blocked on Win11 clients) - [ScreenConnect custom-property slots](reference_screenconnect_custom_property_slots.md) — CP1=Company CP2=Site CP3=Department CP4=Device Type CP8=Tag (API hides labels; UpdateSessionCustomProperties replaces the whole array) - [ScreenConnect cleanup uses wiki as source](feedback_screenconnect_cleanup_wiki_source.md) — per-client SC/RMM metadata cleanup pulls machine->dept/location from the client wiki; enrich the wiki when missing +- [TECH03L systemprofile shortcut corruption](project_tech03l_systemprofile_shortcut_corruption.md) — Auto-Claude "opens then closes" = .lnk pointing at nonexistent systemprofile path; repoint, do not debug the app diff --git a/.claude/memory/project_tech03l_systemprofile_shortcut_corruption.md b/.claude/memory/project_tech03l_systemprofile_shortcut_corruption.md new file mode 100644 index 00000000..57887037 --- /dev/null +++ b/.claude/memory/project_tech03l_systemprofile_shortcut_corruption.md @@ -0,0 +1,23 @@ +--- +name: tech03l-systemprofile-shortcut-corruption +description: ACG-TECH03L shortcuts got rewritten to nonexistent systemprofile paths (Auto-Claude "opens then closes"); repointed via RMM 2026-07-04 +metadata: + type: project +--- + +On ACG-TECH03L (Howard's laptop, "tech-03"), the desktop + Start Menu **Auto-Claude** +shortcuts and the **UltraSearch** shortcut were found pointing at +`C:\Windows\system32\config\systemprofile\...` paths that do not exist — symptom was +"ClaudeTools opens and then closes" when Howard double-clicked the launcher. The Claude +Code CLI itself was healthy (2.1.160, node v24.16.0, repo + identity.json intact). + +**Why:** something (likely an installer/updater running under SYSTEM context, e.g. via +RMM) resolved `%LOCALAPPDATA%`/`%USERPROFILE%` to the SYSTEM profile when writing the +shortcuts. If a Claude-adjacent launcher "flashes and closes" on a fleet machine, check +the .lnk TargetPath for `systemprofile` FIRST before debugging the app. + +**How to apply:** repoint the .lnk to the real per-user install +(`C:\Users\\AppData\Local\Programs\auto-claude-ui\Auto-Claude.exe`) via +WScript.Shell as SYSTEM — fixed 2026-07-04 (RMM cmd 0c9180df). UltraSearch was left +broken: no user-profile copy exists on disk; needs reinstall if Howard wants it. +Avoid running per-user app installers/updaters from SYSTEM context. diff --git a/errorlog.md b/errorlog.md index 4e4b77b2..d4be5cab 100644 --- a/errorlog.md +++ b/errorlog.md @@ -19,6 +19,12 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure · +2026-07-04 | Howard-Home | rmm/claude-reinstall | fresh claude-code 2.1.201 npm install on ACG-Tech03L crashed 0xC0000005 on first --version run, succeeded on retry; suspected Datto EDR first-touch scan of new binaries [ctx: machine=ACG-Tech03L cmd=9770a7af] + +2026-07-04 | Howard-Home | rmm/ps-encoded | ps-encoded.sh failed on Howard-Home: iconv not found in Git Bash; fell back to powershell.exe base64 encoding [ctx: machine=Howard-Home script=.claude/scripts/ps-encoded.sh] + +2026-07-04 | Howard-Home | ps-encoded | encode produced empty output [ctx: src=C:UsersHowardAppDataLocal/Temp/claude/C--claudetools/2ab704e1-28dd-48fa-81dd-7c4d52d0cf3d/scratchpad/tech03-recon.ps1] + 2026-07-04 | Howard-Home | context-loading/dataforth | [correction] grepped for 'datforth' (user misspelling), found nothing, then found wiki hits on second grep but didn't read wiki/clients/dataforth.md or projects/dataforth-dos.md; user had to say 'use the wiki'. Correct: on any client-name trigger, fuzzy-match spelling and READ the wiki article before asking the user for infra facts 2026-07-04 | Howard-Home | screenconnect/sc-cleanup | [friction] burned many tokens iterating the SERVER/Accounting remap because each WAN-map rebuild re-queried SERVER-named machines (returning all fleet SERVER sessions), re-contaminating; fix = build WAN map ONLY from unique-named (len==1) machines, never from shared names diff --git a/wiki/clients/dataforth.md b/wiki/clients/dataforth.md index b90b9bb6..cbf83a76 100644 --- a/wiki/clients/dataforth.md +++ b/wiki/clients/dataforth.md @@ -2,7 +2,7 @@ type: client name: dataforth display_name: Dataforth Corporation -last_compiled: 2026-06-23 +last_compiled: 2026-07-04 compiled_by: Howard-Home/claude-main sources: - clients/dataforth/docs/overview.md @@ -72,6 +72,12 @@ sources: - .claude/memory/project_ad2_dataforth_fork.md - .claude/memory/ad2-ssh-mtu-blackhole.md - .claude/memory/ad2-comms-via-sync-only.md + - clients/dataforth/session-logs/2026-06/2026-06-23-mike-pbx-no-inbound-calls-fix.md + - clients/dataforth/session-logs/2026-06/2026-06-25-howard-dforth-ship-tdr-bsod.md + - clients/dataforth/session-logs/2026-07/2026-07-01-mike-dataforth-test-data-chain-audit.md + - clients/dataforth/session-logs/2026-07/2026-07-04-howard-mydata-tpsys-smt-controller-access.md + - clients/dataforth/docs/audits/2026-07-01-test-data-chain-audit-AD2.md + - .claude/memory/reference_rmm_spawn_headless_claude.md backlinks: - projects/dataforth-dos - systems/jupiter @@ -79,7 +85,7 @@ backlinks: # Dataforth Corporation -Signal conditioning / data acquisition manufacturer in Tucson, AZ. Long-standing ACG client. Active managed relationship — monthly prepaid block. Notable for 64 MS-DOS 6.22 test stations, a major security incident in March 2026, an ongoing test datasheet pipeline modernization project, an incomplete 2025 post-ransomware recovery restore that silently dropped files across multiple shares (active audit underway), and a new shares/permissions remediation project (Phase 1 still pending client input; a Phase 2 target-state strawman was drafted 2026-06-22). +Signal conditioning / data acquisition manufacturer in Tucson, AZ. Long-standing ACG client. Active managed relationship — monthly prepaid block. Notable for 64 MS-DOS 6.22 test stations, a major security incident in March 2026, an ongoing test datasheet pipeline modernization project, an incomplete 2025 post-ransomware recovery restore that silently dropped files across multiple shares (active audit underway), a shares/permissions remediation project (Phase 1 still pending client input), and — newly documented as of 2026-07-04 — a previously-undocumented legacy Linux SMT line controller (MYDATA TPSys, Fedora Core 3) discovered on the manufacturing VLAN. --- @@ -105,10 +111,10 @@ Signal conditioning / data acquisition manufacturer in Tucson, AZ. Long-standing - **External distributor:** Ginger (gy@quatronix-cn.com) — Quatronix China; receives datasheets - **Billing rate:** Prepaid block; all invoices show $0.00 — hours drawn from block -- **Hours remaining:** 31.5 hrs as of 2026-06-23 (live-check Syncro before billing — `GET /customers/578095`) +- **Hours remaining:** 30.0 hrs as of 2026-07-04 (live-check Syncro before billing — `GET /customers/578095`) - **Syncro customer ID:** 578095 - **Syncro managed assets:** 50 -- **Open Syncro tickets:** 0 as of 2026-06-23 +- **Open Syncro tickets:** 0 as of 2026-07-04 - **Invoice CC:** jantar@dataforth.com --- @@ -120,17 +126,18 @@ Signal conditioning / data acquisition manufacturer in Tucson, AZ. Long-standing | Host | IP | Role | OS | Notes | |---|---|---|---|---| | AD1 | 192.168.0.27 | Primary DC, DNS, FSMO roles, Engineering share | Windows Server 2016 | C:\ at **90%** capacity (C:\Engineering = 787 GB) — critical risk. FSMO roles (assumed all). GuruRMM agent `bf7bc5ee-4167-4a62-912a-c88b11a5943d`. Image plan (`Image2025`) + Files plan (NBF, daily 2 AM, 180-day retention — created 2026-06-05). | -| AD2 | 192.168.0.6 | Secondary DC, TestDataDB service host, NAS mirror, WebShare | Windows Server 2022 | Hosts testdatadb Node.js service on :3000. Wiped by crypto attack 2025 — rebuilt. Windows Firewall disabled (all profiles). Shares: `C:\Shares\{c-drive,e-drive,webshare,test}`. Old `D:\c-drive` data volume is GONE — D: is now a mounted Windows install ISO. MSP360 agent at `C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe`; storage account `ACG-Dataforth`. GuruRMM agent `cfa93bb6-0cdc-4d4e-a29e-1609cda6f047`. No shadow copies. Runs ClaudeTools on `ad2` branch (coord-API isolated; comms via git sync only). | +| AD2 | 192.168.0.6 | Secondary DC, TestDataDB service host, NAS mirror, WebShare | Windows Server 2019 (10.0.17763) | Hosts testdatadb Node.js service on :3000. Wiped by crypto attack 2025 — rebuilt. Windows Firewall disabled (all profiles). Shares: `C:\Shares\{c-drive,e-drive,webshare,test}`. Old `D:\c-drive` data volume is GONE — D: is now a mounted Windows install ISO. MSP360 agent at `C:\Program Files\Arizona Computer Guru\Online Backup\cbb.exe`; storage account `ACG-Dataforth`. GuruRMM agent `cfa93bb6-0cdc-4d4e-a29e-1609cda6f047`. No shadow copies. Runs ClaudeTools on `ad2` branch (coord-API isolated; comms via git sync only). **New (2026-07-01): its GuruRMM agent can also be used to spawn a headless `claude -p` for live read-only ground truth** — see [RMM-Spawned Claude](#rmm-spawned-claude-on-ad2) below; console user `sysadmin`, `claude.exe` v2.1.181 at `C:\Users\sysadmin\.local\bin\`, node v20.10.0. OS build corrected 2026-07-01 (was previously logged as 2022; live audit confirmed Server 2019). | | FILES-D1 | 192.168.0.189 | File server | Windows Server 2016 | Shares: `E:\Shares\{sales,archive}`. GuruRMM agent `8566a19d-49a9-4f8b-9c6c-012cc934484b`. **NOTE: `staff` share is missing** on FILES-D1 — separate issue. | | SAGE-SQL | 192.168.0.153 | Sage ERP (S:), RDS Session Host/Connection Broker/Web Access | Windows Server 2016 | RDS licensing grace period was expired (reset 2026-05-06). TSGateway disabled (server not externally exposed). New self-signed RDS cert installed. Bitdefender GravityZone managed AV. Share: `C:\sage`. GuruRMM agent `120ba7bf-8544-48a0-98a1-40ed5cdd3e1f`. | | 3CX | 192.168.0.125 | Phone system (possibly inactive) | — | Last logon Oct 2025. Production phones live on VLAN 100 under the Sangoma/FreePBX PBX — 3CX role likely superseded. | | DF-HYPERV-B | 192.168.0.123 | Hyper-V hypervisor | Windows Server 2025 | GuruRMM enrolled. Newest server in environment. VM inventory not captured. | | DF-SVR-D2-Sync | — | (role TBD) | — | GuruRMM enrolled | | ENG-DEV-SERVER | 192.168.0.126 | Engineering dev server | Windows 11 Pro | GuruRMM enrolled | -| D2TESTNAS | 192.168.0.9 | SMB1 bridge for DOS test stations + AOI XP backup; Neptune Exchange colocation routing | Debian 13 (trixie), Samba 4.22.6 | **Repurposed Netgear ReadyNAS.** SMB1 enabled globally (CORE..SMB3, NTLMv1) — required for DOS 6.22 stations. rsync daemon on port 873 (module `test`, user `rsync`, hosts allow 192.168.0.0/24 + 172.16.0.0/12). SSH: `root@192.168.0.9`. Tailscale route for 172.16.0.0/22. **Shares:** `test`/`datasheets`/`snapshots` (guest; `hosts deny 192.168.1.175`), `aoibackup` (XP-only — see Access). Acts as jump host for UDM SSH (D2TESTNAS direct-tcpip channel to 192.168.0.254). | +| D2TESTNAS | 192.168.0.9 | SMB1 bridge for DOS test stations + AOI XP backup; Neptune Exchange colocation routing; rsync source/target for the test-data pipeline | Debian 13 (trixie), Samba 4.22.6 | **Repurposed Netgear ReadyNAS.** SMB1 enabled globally (CORE..SMB3, NTLMv1) — required for DOS 6.22 stations. rsync daemon on port 873 (module `test`, user `rsync`, hosts allow 192.168.0.0/24 + 172.16.0.0/12). SSH: `root@192.168.0.9`. Tailscale route for 172.16.0.0/22. **Shares:** `test`/`datasheets`/`snapshots` (guest; `hosts deny 192.168.1.175`), `aoibackup` (XP-only — see Access). Acts as jump host for UDM SSH (D2TESTNAS direct-tcpip channel to 192.168.0.254). **2026-07-01 audit:** root SSH key auth from AD2 (`root@192.168.0.9`) is broken (publickey denied) — no operational impact since the live sync uses the rsync daemon, not SSH, but the dormant SCP fallback script would fail if re-enabled (F9). | | ESXi hosts | 192.168.0.122, 192.168.0.124 | VMware ESXi hypervisors | ESXi | — | -| UDM Firewall | 192.168.0.254 | Perimeter firewall/router | UniFi OS 5.1.15 | MAC d0:21:f9:6c:11:02. Also responds on 192.168.0.1. SSH: `azcomputerguru@192.168.0.254`, root SSH key added 2026-06-08, 2FA push required. Vault: `clients/dataforth/udm.sops.yaml`. C2 IPs blocked via iptables (NOT permanent — need to add to UniFi UI). Boot scripts in `/data/on_boot.d/`: `10-neptune-snat.sh` (Neptune outbound SNAT), `30-freepbx-sip-forward.sh` (SIP DNAT, WAN UDP 5060 source-locked to 66.7.123.0/24 → 192.168.100.2; SIP-only — do NOT add RTP forward). | -| PBX (Sangoma FreePBX) | 192.168.100.2 | VoIP PBX — production phones on 192.168.100.0/24 | Sangoma FreePBX 17 / Asterisk 22.5.2 | FirstDigital PJSIP trunk; SBC 66.7.123.215:5060 (Sonus), match 66.7.123.0/24; IP-auth (no registration). `qualify_frequency=0` (FD SBC ignores OPTIONS — do NOT revert). TFTP provisioning for Cisco SPA502G phones. SSH: `sangoma@192.168.100.2`. Vault: `clients/dataforth/pbx.sops.yaml`. [WARNING] Re-apply `PJSip.class.php` line-504 patch after any `fwconsole ma updateall`. | +| UDM Firewall | 192.168.0.254 | Perimeter firewall/router | UniFi OS 5.1.15 | MAC d0:21:f9:6c:11:02. Also responds on 192.168.0.1. SSH: `azcomputerguru@192.168.0.254`, root SSH key added 2026-06-08, 2FA push required. Vault: `clients/dataforth/udm.sops.yaml`. C2 IPs blocked via iptables (NOT permanent — need to add to UniFi UI). Boot scripts in `/data/on_boot.d/`: `10-neptune-snat.sh` (Neptune outbound SNAT), `30-freepbx-sip-forward.sh` (SIP DNAT, WAN UDP 5060 source-locked to 66.7.123.0/24 → 192.168.100.2; SIP-only — do NOT add RTP forward). **[WARNING] Confirmed 2026-06-23: the SIP DNAT rule can be silently flushed by a UniFi controller provision/update, not only a reboot** — the on_boot.d script only re-applies at boot, so a mid-uptime provision event leaves inbound calls dead until the script is manually re-run. Recommend adding a persistent UI port-forward rule as a belt-and-suspenders measure (still not done as of 2026-07-04). | +| PBX (Sangoma FreePBX) | 192.168.100.2 | VoIP PBX — production phones on 192.168.100.0/24 | Sangoma FreePBX 17 / Asterisk 22.5.2, Debian 12 | FirstDigital PJSIP trunk; SBC 66.7.123.215:5060 (Sonus), match 66.7.123.0/24; IP-auth (no registration). `qualify_frequency=0` (FD SBC ignores OPTIONS — do NOT revert). TFTP provisioning for Cisco SPA502G phones. Extensions 201-343. SSH: `sangoma@192.168.100.2` (ACG SSH key also authenticates). Vault: `clients/dataforth/pbx.sops.yaml` — password corrected 2026-06-23 (prior entry had a backslash-escaping corruption in the stored value; re-verify the vault entry byte-for-byte before use rather than assuming it is stale). [WARNING] Re-apply `PJSip.class.php` line-504 patch after any `fwconsole ma updateall`. **NOTE:** `sangoma` user is in the `sudo` group but sudo authorization on this box appears not actually granted — verify before assuming privileged ops will work via sudo. | +| **MYDATA TPSys SMT Controller** (`myserver`) | 192.168.1.x (verify — exact IP unconfirmed) | MYDATA/Mycronic TPSys pick-and-place SMT production-line controller | Fedora Core 3 "Heidelberg" (Nov 2004), kernel 2.6.16.20, glibc ~2.3.5, bash 3.00, **LILO** bootloader, **SysV init** (no systemd) | **NEW, discovered 2026-07-04.** On VLAN 2 "mydata" (192.168.1.0/24, gateway 192.168.1.1). TPSys operator UI runs under X (runlevel 5); local PostgreSQL (uid 500) backs TPSys. Accounts: `root`, `tpsys` (TPSys app user), `tpspool` (TPSys spool), `postgres`. **No credential existed anywhere (vault or wiki) prior to this discovery** — root password was RESET via physical-console LILO recovery (no prior password to lose) and is now vaulted at `clients/dataforth/mydata-smt.sops.yaml` (reference the vault path only; never the raw password). `tpsys` being added to the `wheel` group plus a scoped `NOPASSWD` sudoers entry for the app-launch command (directed, not yet verified as landed). **GuruRMM agent CANNOT run on this box** — see Patterns below. | **Neptune Exchange (ACG infrastructure, physically at Dataforth D2):** - `neptune.acghosting.com` | internal `172.16.3.11` | external inbound `67.206.163.124` / outbound `67.206.163.122` @@ -141,13 +148,19 @@ Signal conditioning / data acquisition manufacturer in Tucson, AZ. Long-standing - Vault: `clients/dataforth/neptune-exchange.sops.yaml` - [WARNING] TODO: Resubnet Dataforth UDM to a non-overlapping range to permanently fix Neptune routing +### RMM-Spawned Claude on AD2 + +- **New capability proven 2026-07-01:** a headless `claude -p` can be launched on AD2 through its GuruRMM agent (`cfa93bb6-...`, `context:user_session`, since `sysadmin` is logged into the console and the RMM elevated token works), detached (`Start-Process ... -WindowStyle Hidden`) so it survives the RMM command-timeout window, writing a deliverable file + `DONE.txt` marker that a background poller watches. This is a live read-only ground-truth channel that works despite AD2 being isolated from the ACG coord API (172.16.3.30 unreachable from Dataforth LAN) — it does NOT replace the git-sync handoff for anything that needs to write back to the shared repo. +- **Gotcha:** a stale machine-level `ANTHROPIC_API_KEY` (108 chars, invalid) on AD2 shadows `sysadmin`'s working OAuth credentials (`C:\Users\sysadmin\.claude\.credentials.json`). Must `Remove-Item Env:\ANTHROPIC_API_KEY` before invoking `claude -p`, or it fails with `Invalid API key`. +- Reference: `.claude/memory/reference_rmm_spawn_headless_claude.md`; full pattern + transcript in `clients/dataforth/session-logs/2026-07/2026-07-01-mike-dataforth-test-data-chain-audit.md`. + ### Share -> Server -> Physical Path Map | Drive/Share | Server | Physical path | Notes | |---|---|---|---| | Q: / `c-drive` | AD2 | `C:\Shares\c-drive` | Old `D:\c-drive` is gone (D: = mounted install ISO) | -| T: / `e-drive` | AD2 | `C:\Shares\e-drive` | — | -| X: / `webshare` | AD2 | `C:\Shares\webshare` | — | +| T: / `e-drive` | AD2 | `C:\Shares\e-drive` | On AD2 itself, `T:` is `\\ad2\e-drive` — NOT the NAS. DOS stations separately map their own `T:` to `\\D2TESTNAS\test` (see DOS Test Station Data Pipeline pattern). | +| X: / `webshare` | AD2 | `C:\Shares\webshare` | On AD2, `X:` = `\\ad2\webshare`. DOS stations separately map their own `X:` to `\\D2TESTNAS\datasheets`. | | S: / `sage` | SAGE-SQL | `C:\sage` | — | | W: / `sales` | FILES-D1 | `E:\Shares\sales` | — | | Y: / `archive` | FILES-D1 | `E:\Shares\archive` | — | @@ -162,9 +175,11 @@ Signal conditioning / data acquisition manufacturer in Tucson, AZ. Long-standing | Engineering | ~12 | Win 10/11 Pro | HGHAUBNER (192.168.0.148) — Georg's PC; `D:` = full pre-attack backup of all 7 DF shares (`DF C-Drive`, `DF E-Drive`, `DF WebShare`, `DF Sage`, `DF Server Sales/Archive/Engineering`, + personal). GuruRMM agent `2aefe0d5-2357-4bdd-965a-abfccb4767a5`. D1-PWRM for PWRM10 test. | | Manufacturing/Assembly | ~14 | Win 10/11 Pro | AS24, AS26 + various assembly/hi-pot stations | | Office/Admin | ~12 | Win 10/11 Pro | DF-GAGETRAK (192.168.0.102) — GAGEtrak calibration host. DF-JOEL2 (192.168.0.174) — compromised 2026-03-27, remediated. | +| Shipping | (part of Office/Admin count) | Win 10/11 Pro | **DFORTH-Ship** — GuruRMM agent `db17e069-2948-4cbc-97ea-1da721edcaf5`, HP EliteDesk 800 G1 USDT (BIOS 2014-12-10, ~11.5 yrs old). Recurring BSOD `0x116 VIDEO_TDR_FAILURE` on integrated Intel HD Graphics 4600 — see Patterns. Do not confuse with near-twin host **DForth-Shipp** (`95991b45-d843-4586-8275-9996d0d9ae17`), a separate machine. | | End-of-Life (Win 7) | 3 | Windows 7 Pro | LABELPC (192.168.0.100), LABELPC2 (192.168.0.98), D2-RCVG-003 (192.168.0.47) — EOL, on network | | AOI Optical Inspection (XP) | 1 | Windows XP | WinXPBE-724667 @ **192.168.1.175** on VLAN 2 (mydata/SMT). Holds the AOI machine's external drive; backs up to `\\192.168.0.9\aoibackup` (SMB1, XP-only). EOL. See AOI runbook + 2026-06-01 session log. | -| DOS Test Stations | 64 | MS-DOS 6.22 | TS-1 through TS-30 + variants. Not domain-joined. SMB1 via D2TESTNAS. | +| SMT Line Controller (legacy Linux appliance) | 1 | Fedora Core 3 "Heidelberg" (2004) | **MYDATA TPSys** (`myserver`), on VLAN 2 (mydata/SMT), ~20 years old. Full detail in Infrastructure Servers table above. No GuruRMM agent possible (glibc/kernel/init all below the agent's floor) — agentless monitoring planned. | +| DOS Test Stations | 64 | MS-DOS 6.22 | TS-1 through TS-30 + variants (62 `TS-*` folders confirmed on NAS 2026-07-01). Not domain-joined. SMB1 via D2TESTNAS. See DOS Test Station Data Pipeline pattern for the 2026-07-01 audit findings. | ### Email & Identity @@ -196,6 +211,7 @@ Signal conditioning / data acquisition manufacturer in Tucson, AZ. Long-standing - **Firewall/Router:** UniFi Dream Machine Pro at 192.168.0.254 (also 192.168.0.1), UniFi OS 5.1.15 - **Network:** Flat (no VLANs on main LAN — 192.168.0.0/24). Voice/PBX VLAN: 192.168.100.0/24 — production phones live here. **VLAN 2 "mydata" (192.168.1.0/24)** = SMT production-line network (gateway 192.168.1.1); members on the *D2-SMT Switch* (USW Enterprise 8) + *D2-Breakroom* port 12. Supersedes the earlier note that 192.168.1.0/24 was an unused UDM default voice VLAN — it is in active use by SMT. Inter-VLAN routing from mydata → main LAN is currently OPEN. - **mydata members (2026-06-01):** WinXPBE-724667 (AOI XP, .175), goldstar19, DESKTOP-FT0T4MK, My9-PC, + 3 unnamed industrial/SMT devices (MAC 00:90:fb:80:f0:c6, 00:80:79:05:23:f2, 00:80:79:04:47:e7). + - **mydata addition (2026-07-04):** **MYDATA TPSys controller** (`myserver`) confirmed present on this VLAN — hostname distinct from the named members above; exact IP still to confirm (verify). Likely corresponds to one of the previously "unnamed industrial" devices or is a device not yet captured in the member list; reconcile on next VLAN sweep. - **VPN:** OpenVPN for ACG remote access. Client subnet 192.168.6.x (GURU-5070 gets 192.168.6.2). [WARNING] GURU-5070 OpenVPN adapter "Local Area Connection" (ifIndex 12) MTU must be set to 1400 — default 1500 causes PMTU blackhole (tunnel path MTU ~1424; bulk SSH/SCP silently drops). Verify/re-apply: `Set-NetIPInterface -InterfaceIndex 12 -AddressFamily IPv4 -NlMtuBytes 1400`. Permanent fix: add `mssfix 1360` server-side on the Dataforth OpenVPN server. - **Drive mappings (GPO):** B: (\\ad1\itsvc), Q: (\\ad2\c-drive), S: (\\SAGE-SQL\sage), T: (\\ad2\e-drive), W: (\\files-d1\sales), X: (\\ad2\webshare), Y: (\\files-d1\archive). DOS test stations: T: (\\D2TESTNAS\test), X: (\\D2TESTNAS\datasheets) @@ -203,8 +219,9 @@ Signal conditioning / data acquisition manufacturer in Tucson, AZ. Long-standing - **Site name:** Dataforth D1 | Site ID: `3a2f6866-26cd-452c-9806-a8df21475c3c` - **Site API key:** vault `clients/dataforth/...` [check vault for current entry] -- **Fleet size:** 45 agents enrolled as of 2026-06-04; Syncro managed count 50 as of 2026-06-19 +- **Fleet size:** 45 agents enrolled as of 2026-06-04; Syncro managed count 50 as of 2026-07-04 - **[WARNING] GuruRMM enrollment workaround:** WebSocket auth in `ws/mod.rs` does not validate `enrolled_agents.agent_key_hash`. New agent installs must overwrite registry AgentKey with the site API key (not the enrollment AgentKey) and restart service. See Gitea issue #8. +- **[WARNING] Agent floor confirmed 2026-07-04 (MYDATA TPSys case):** the Linux installer (`agent/scripts/install.sh`) requires glibc ~2.17+, kernel >=2.6.32, and a systemd host. Legacy Linux appliances below that floor (e.g. Fedora Core 3, SysV init) cannot run the agent — plan agentless monitoring (ICMP/TCP probe, SSH heartbeat from a reachable host) for such boxes instead of attempting install. **Known enrolled agents:** @@ -212,14 +229,21 @@ Signal conditioning / data acquisition manufacturer in Tucson, AZ. Long-standing |---|---|---| | DF-GAGETRAK | `7626d82c-0736-47a6-8bc6-68e39859caed` | Enrolled 2026-04-23 (auth workaround applied) | | HGHAUBNER | `2aefe0d5-2357-4bdd-965a-abfccb4767a5` | Georg's PC; pre-attack backup on D: | -| AD2 | `cfa93bb6-0cdc-4d4e-a29e-1609cda6f047` | Enrolled 2026-06-04 | +| AD2 | `cfa93bb6-0cdc-4d4e-a29e-1609cda6f047` | Enrolled 2026-06-04; also used 2026-07-01 to spawn headless Claude for the test-data-chain audit | | AD1 | `bf7bc5ee-4167-4a62-912a-c88b11a5943d` | Enrolled 2026-06-04 | | FILES-D1 | `8566a19d-49a9-4f8b-9c6c-012cc934484b` | Enrolled 2026-06-04 | | SAGE-SQL | `120ba7bf-8544-48a0-98a1-40ed5cdd3e1f` | Enrolled 2026-06-04 | | DF-HYPERV-B | (see RMM dashboard) | Enrolled 2026-06-04 | | DF-SVR-D2-Sync | (see RMM dashboard) | Enrolled 2026-06-04 | | ENG-DEV-SERVER | (see RMM dashboard) | Enrolled 2026-06-04 | -| (37 additional agents) | — | Mix of workstations; full list in GuruRMM dashboard | +| DFORTH-Ship | `db17e069-2948-4cbc-97ea-1da721edcaf5` | Shipping-station PC; recurring TDR BSOD — see Patterns | +| (36 additional agents) | — | Mix of workstations; full list in GuruRMM dashboard | + +**Cannot be enrolled:** + +| Host | Reason | +|---|---| +| MYDATA TPSys (`myserver`) | Fedora Core 3: glibc ~2.3.5 (<2.17 floor), kernel 2.6.16 (<2.6.32 floor), SysV init (no systemd unit target) — three independent hard blockers. Confirmed 2026-07-04. | ### Backup Architecture @@ -234,18 +258,19 @@ Signal conditioning / data acquisition manufacturer in Tucson, AZ. Long-standing | Application | Host | URL/Port | Notes | |---|---|---|---| -| TestDataDB | AD2 | http://192.168.0.6:3000 | Node.js + Express, PostgreSQL 18, 469K records. Internal LAN only. Redesigned UI deployed 2026-06-18 (cert-fit, publish chips, push toasts, full-screen results). | +| TestDataDB | AD2 | http://192.168.0.6:3000 | Node.js + Express, PostgreSQL 18.3. Internal LAN only. Redesigned UI deployed 2026-06-18 (cert-fit, publish chips, push toasts, full-screen results). Row counts per the 2026-07-01 audit: `test_records` 475,553, `work_orders` 34,149, `work_order_lines` 64,051; 99.3% of test_records have `api_uploaded_at` set (HTTP API uploader is the live web-delivery path — the older `For_Web`/ASP.NET path has been dead since 2026-05-11). | | Sage ERP | SAGE-SQL | \\SAGE-SQL\sage (S:) | RDS-served RemoteApp | | GageTrak | DF-GAGETRAK (192.168.0.102) | — | Calibration tracking. Sends email via calibration@dataforth.com (SMTP). GuruRMM enrolled. | | Dataforth Product API | Hoffman's servers | https://www.dataforth.com/api/v1/TestReportDataFiles | OAuth2 client_credentials. Vault: `clients/dataforth/api-oauth.sops.yaml`. Used actively to recover DSCA33/45 and 8B/5B/SCM spec templates. | -| QuickBASIC 4.5 ATE | 64 DOS stations | T:\ (\\D2TESTNAS\test) | Automated test equipment programs. 1,470+ product model specs. | +| QuickBASIC 4.5 ATE | 64 DOS stations | T:\ (\\D2TESTNAS\test) | Automated test equipment programs. 1,470+ product model specs. Inbound spec/software distribution to stations is currently broken (root-caused 2026-07-01 — see DOS Test Station Data Pipeline pattern and Syncro #32489). | | Power Monitor SPA | Georg's dev / TBD | — | Vanilla-JS SPA for Dataforth power meters (built by Georg/Antigravity AI). Demo at PWM.dataforth.com proposed; gateway architecture designed. Parked pending Mike↔Georg conversation. `clients/dataforth/power-monitor-demo/` | +| MYDATA TPSys | MYDATA TPSys controller (`myserver`, VLAN 2) | local X UI / local PostgreSQL | SMT pick-and-place line control software. Newly documented 2026-07-04 — see Infrastructure Servers table. | --- ## Syncro Asset Inventory (2026-06-02 Reconciliation) -Pulled full Syncro asset list for customer_id `578095`: **78 assets** across 2 pages. Syncro currently shows 50 managed assets (2026-06-19 live data); reconciliation/cleanup ongoing. +Pulled full Syncro asset list for customer_id `578095`: **78 assets** across 2 pages. Syncro currently shows 50 managed assets (confirmed again 2026-07-04 live pull); reconciliation/cleanup ongoing. ### Reconciliation Result @@ -313,7 +338,7 @@ Syncro asset IDs: 23845, 149614, 9708445, 9357407, 9276901, 9212922, 9078651, 88 ### Domain / Server Access - **AD2 SSH:** `ssh sysadmin@192.168.0.6` (port 22) — vault: `clients/dataforth/ad2.sops.yaml` → `credentials.password` — NOTE: stale backslash escape in vault entry; strip with `sed 's/\\//g'`. MTU-sensitive: GURU-5070 OpenVPN adapter ifIndex 12 must be MTU 1400 for reliable bulk transfers. - **AD1 SSH:** `ssh sysadmin@192.168.0.27` — vault: `clients/dataforth/ad1.sops.yaml` -- **D2TESTNAS SSH:** `ssh root@192.168.0.9` — vault: `clients/dataforth/d2testnas.sops.yaml`. Use root, NOT sysadmin (sysadmin SSH fails on D2TESTNAS). SSH key from acg-guru-5070 authorized. +- **D2TESTNAS SSH:** `ssh root@192.168.0.9` — vault: `clients/dataforth/d2testnas.sops.yaml`. Use root, NOT sysadmin (sysadmin SSH fails on D2TESTNAS). SSH key from acg-guru-5070 authorized. **From AD2, root SSH key auth to D2TESTNAS is currently broken** (publickey denied, confirmed 2026-07-01) — the live rsync-daemon sync path does not depend on it, but fix or retire the dormant SCP script that does. - **D2TESTNAS `aoibackup` share (AOI XP backup):** `\\192.168.0.9\aoibackup` — Samba user `admin` (password matches the XP's local login), `hosts allow = 192.168.1.175` only, `browseable = no`. Other NAS shares explicitly deny 192.168.1.175. Creds in vault: `clients/dataforth/d2testnas.sops.yaml → credentials.smb.aoi-user` / `.aoi-password` / `.aoi-share`. - **UDM SSH:** `ssh azcomputerguru@192.168.0.254` (2FA push) or `ssh root@192.168.0.254` (root SSH key installed 2026-06-08). Jump via D2TESTNAS: paramiko `direct-tcpip` channel or ProxyJump. Vault: `clients/dataforth/udm.sops.yaml` (corrected 2026-06-09). - **SAGE-SQL SSH:** `ssh sysadmin@192.168.0.153` — SSH key (`C:\ProgramData\ssh\administrators_authorized_keys` on SAGE-SQL) @@ -321,6 +346,12 @@ Syncro asset IDs: 23845, 149614, 9708445, 9357407, 9276901, 9212922, 9078651, 88 - **WinRM (AD2/AD1):** port 5985 — pywinrm with NTLM, user `INTRANET\sysadmin` - **HGHAUBNER:** No SSH. Reached via GuruRMM agent `2aefe0d5`. Logged-in user `intranet\ghaubner`. Cross-machine file writes use existing GPO-mapped drives only (Q: → \\ad2\c-drive, T: → \\ad2\e-drive, etc.). +### MYDATA TPSys SMT Controller (new, 2026-07-04) +- **Root:** password — vault `clients/dataforth/mydata-smt.sops.yaml`. Console/network access method beyond physical console not yet documented (verify — likely SSH once IP is confirmed). Exact IP on VLAN 2 "mydata" (192.168.1.0/24) not yet confirmed (verify). +- **Recovery method if locked out again:** at the LILO `boot:` prompt, boot `linux init=/bin/bash rw` to land in a passwordless root shell (bypasses `sulogin`, which on this Red-Hat-family box would otherwise demand the root password even in single-user mode), then `mount -o remount,rw /` and `passwd root`. Reboot with `reboot -f` or, if that hangs, `echo b > /proc/sysrq-trigger`. +- **Accounts:** `root`, `tpsys` (TPSys application user, being added to `wheel` + scoped `NOPASSWD` sudo for the app-launch command), `tpspool` (TPSys spool user), `postgres` (uid 500, local TPSys database). +- **No prior credential existed** for this machine in vault or wiki before 2026-07-04 — it was undocumented until discovered at the physical console. + ### M365 / Entra - **M365 admin:** sysadmin@dataforth.com — vault: `clients/dataforth/m365.sops.yaml` - **Tenant ID:** `7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584` @@ -346,7 +377,7 @@ Syncro asset IDs: 23845, 149614, 9708445, 9357407, 9276901, 9212922, 9078651, 88 - ESXi-124: 192.168.0.124 — vault: `clients/dataforth/esxi-124.sops.yaml` ### PBX -- Vault: `clients/dataforth/pbx.sops.yaml` +- Vault: `clients/dataforth/pbx.sops.yaml` (password corrected 2026-06-23 — see Infrastructure Servers table) - SSH: `sangoma@192.168.100.2` --- @@ -369,12 +400,13 @@ Syncro asset IDs: 23845, 149614, 9708445, 9357407, 9276901, 9212922, 9078651, 88 ### Voice / Phones / FreePBX - **Production phones VLAN:** 192.168.100.0/24. PBX at .196 / .2. All production phones live here. -- **Unifi default voice VLAN (192.168.1.0/24):** NOT used for production — phones landing here cannot reach PBX. Switch port misconfiguration symptom: phone shows wrong date/time (NTP failure) and no dial tone. +- **Unifi default voice VLAN (192.168.1.0/24):** NOT used for production voice — phones landing here cannot reach PBX. Switch port misconfiguration symptom: phone shows wrong date/time (NTP failure) and no dial tone. (This subnet is however in active use for the mydata/SMT VLAN — see Network section; do not confuse the two purposes.) - **D1-Server-Room port 1:** Controls lobby drop → must stay on VLAN 100. Reverted to default once before (2026-05-04 incident). - **FirstDigital trunk — `qualify_frequency=0`:** FD's Sonus SBC ignores SIP OPTIONS keepalives. Setting `qualify=0` in the `pjsip` DB (id=1) prevents trunk from going Unavailable. **Do NOT revert to a non-zero qualify.** (Total phone outage 2026-06-08 was caused by FD SBC not answering OPTIONS, making trunk go Unavailable and blocking all INVITEs.) - **PJSip.class.php line 504 patch must be re-applied** after any `fwconsole ma updateall`. It is wiped by FreePBX updates. Backup before each update (`PJSip.class.php.bak.`). - **Do NOT port-forward the RTP range (10000-20000)** on the UDM for this trunk. A static RTP DNAT creates a conntrack collision with the PBX's outbound RTP — inbound works but outbound audio dies. SIP 5060 forward only (source-locked to 66.7.123.0/24). Current on_boot.d script (`30-freepbx-sip-forward.sh`) is SIP-only, correct. -- **Inbound SIP relies on `/data/on_boot.d/30-freepbx-sip-forward.sh`** — not a persistent UniFi UI rule. Must survive UDM reboot via the script. Recommend Mike add a UI port-forward as a belt-and-suspenders measure. +- **Inbound SIP relies on `/data/on_boot.d/30-freepbx-sip-forward.sh`** — not a persistent UniFi UI rule. Must survive UDM reboot via the script. **Confirmed 2026-06-23: the risk window is broader than "survives reboot" — a mid-uptime UniFi controller provision/update can also flush the SIP DNAT** while the box never reboots, silently killing inbound calls until the script is manually re-run (`sh /data/on_boot.d/30-freepbx-sip-forward.sh`). Diagnostic: `iptables -t nat -S | grep -iE '5060|192.168.100.2'` on the UDM — empty output means the rule is gone. Recommend Mike add a UI port-forward as a belt-and-suspenders measure (not yet done as of 2026-07-04). +- **PBX sudo authorization gap:** `sangoma` is in the `sudo` group but sudo authorization on this distro appears not actually granted (a sudo-based liveness test can fail on authorization, not auth — do not use it to judge whether a password/credential is stale). ### Exchange Online / Email - **INKY PhishFence StopProcessingRules:** Kills all subsequent transport rules. Use inbox rules for per-mailbox forwarding, NOT transport rules. @@ -385,6 +417,8 @@ Syncro asset IDs: 23845, 149614, 9708445, 9357407, 9276901, 9212922, 9078651, 88 ### GuruRMM Agent Deployment - **WebSocket auth bug (Issue #8):** enrolled_agents.agent_key_hash is never checked by ws/mod.rs. Workaround: after MSI install, overwrite registry `HKLM:\SOFTWARE\GuruRMM\AgentKey` with the site API key (not enrollment AgentKey), then restart service. - **rmm-api.azcomputerguru.com must be grey-clouded** (DNS-only, not proxied) — Cloudflare proxy blocks WebSocket. Do NOT re-enable orange cloud. Gitea Issue #9. +- **No RMM agent possible on FC3-class legacy appliances (confirmed 2026-07-04, MYDATA TPSys):** the Linux agent installer requires glibc ~2.17+, kernel >=2.6.32, and installs a systemd unit. A box on glibc ~2.3.5 / kernel 2.6.16 / SysV init (Fedora Core 3 "Heidelberg", Nov 2004) fails all three floors simultaneously — do not attempt install; plan agentless monitoring (ICMP/TCP probe or SSH heartbeat from a host that can reach the appliance's VLAN, e.g. D2TESTNAS or the RMM server, since inter-VLAN routing from mydata to main LAN is open) instead. If this becomes a recurring need across other legacy appliances, file a `/feature-request` for legacy/appliance Linux monitoring support in GuruRMM. +- **Legacy Red-Hat-family recovery pattern:** if a box like this is ever locked out again with no vaulted credential, LILO `boot:` → `linux init=/bin/bash rw` gives a passwordless root shell and bypasses `sulogin` (which on this OS family would otherwise demand the root password even in single-user mode). Always check BOTH vault and wiki before assuming a machine has no documented credential — a prior session missed the wiki check here and had to be corrected. ### Cross-Machine File Operations (Windows Domain) - **Double-hop / WTS-impersonation blocks fresh UNC paths.** When running commands in GuruRMM `user_session` (or via SSH-through-another-server), the impersonated token carries no network credentials. `net use` and fresh `\\server\share` paths fail with Access Denied. @@ -401,6 +435,7 @@ Syncro asset IDs: 23845, 149614, 9708445, 9357407, 9276901, 9212922, 9078651, 88 - **AD2 operates on the `ad2` git branch.** Fork is rebased from main + thin Dataforth-specific commits. Do NOT edit shared fleet files on `ad2` — conflicts on every sync. Dataforth context lives in `clients/dataforth/CLAUDE.dataforth.md`. - **AD2 is coord-API isolated:** 172.16.3.30 is unreachable from Dataforth LAN. Coord messages, locks, and todos NEVER reach AD2. All inter-session coordination goes through git sync: committed handoff docs + `## Note for ` blocks. Do NOT use the coord skill for AD2. - **sync.sh on AD2:** not fork-aware on the push step (always tries `main`); force-push manually: `git push --force-with-lease origin ad2` after rebasing. +- **New (2026-07-01): AD2's GuruRMM agent can be used as a live read-only ground-truth channel** by spawning a headless `claude -p` through it (`context:user_session`, detached, poll for a `DONE` marker) — bypasses the coord isolation for READ-ONLY investigation without waiting on git-sync round trips. Unset the stale machine-level `ANTHROPIC_API_KEY` env var first or auth fails. See [RMM-Spawned Claude](#rmm-spawned-claude-on-ad2) above. ### Post-Ransomware Recovery Restore (2025) — Incomplete File Migration - **The 10/1/2025 recovery restore was incomplete.** The `Restore plan 10/1/2025` (~3.4M files) migrated each share from the old `D:\` layout to the current `C:\Shares\...` layout on AD2 and dropped files in the process. Proven case: SP1366 MAQ20 Communications Module — each `PRINTOUTS FOR MANUFACTURING` folder for revisions E–H received only one file (the drill panel) when the backup contained ~6 files per revision. The 9/29/2025 file-level backup confirms the files existed before the restore. @@ -416,14 +451,29 @@ Syncro asset IDs: 23845, 149614, 9708445, 9357407, 9276901, 9212922, 9078651, 88 - **Drive-letter strategy — Option A recommended:** keep current Q/S/T/W/Y/B mappings and realize the tree *logically* (reorg folders within each share + apply groups) for the first rollout — lowest disruption, no app/UNC breakage, no retraining. Hold physical consolidation to one `Company` drive (Option B) as a later optional phase after a hard-coded-UNC-path audit (DOS, Sage, datasheet pipeline, GageTrak/Epicor). The permission model is identical either way. - **Strawman is NOT a build order — six items still gate Phase 2 sign-off (need the client):** confirm the inferred department list; the per-department RW/RO/none access matrix; named access for sensitive data (Payroll/OSHA/POs/Accounting — likely HR/Finance sign-off, not just Dan); department rosters to populate groups; legacy cleanup approval (person-named / "Do not use" folders); and an Engineering destination volume (AD1 C: ~90% full blocks any ENGR restructure). +### DOS Test Station Data Pipeline (new, 2026-07-01 ground-truth audit) +- **Root cause of Syncro #32489 confirmed (F1, HIGH):** the deployed inbound spec downloader `T:\COMMON\ProdSW\NWTOC.BAT` v5.0 (mirrored to NAS `COMMON/ProdSW/NWTOC.BAT`) copies only `*.BAT` and `*.EXE` from the NAS to stations — **zero `.DAT` files**. Its own changelog header says so verbatim: "Added EXE copy, removed DATA folder copies (avoid cyclic overwrites)". No version of NWTOC, past or present, has ever distributed the shared `COMMON` master specs — the fix must ADD a data copy, not "restore" one. Engineering masters (e.g. `5BMAIN.DAT`, updated 2026-06-26) reach the NAS fine; they simply never reach the stations. +- **New risk found (F2, HIGH, needs on-station confirmation):** the deployed `NWTOC.BAT` v5.0 and `CTONWTXT.BAT` v2.3 use `COPY /Y`. `/Y` is **not a valid MS-DOS 6.22 switch** (introduced in MS-DOS 7.0/Windows 95); 6.22's `COPY` supports only `/A /B /V`. If the stations run genuine 6.22, `COPY /Y` returns `Invalid switch - /Y` and copies nothing — meaning NWTOC has been silently copying NOTHING (not even the .BAT/.EXE files it's supposed to) since it was deployed 2026-03-16. This was independently confirmed by Grok in verify mode. The pivotal unresolved question — whether the stations run true 6.22 or MS-DOS 7.x — is empirical and can only be checked on a station itself (`VER`, then `COPY /Y NUL C:\TEST.TXT`); stations have no RMM agent. The upload path (`CTONW.BAT` v5.0) is unaffected — it uses plain `COPY` — which is why test data has kept flowing even if NWTOC is dead. +- **Do NOT judge DOS-6.22 compatibility from the root-level `test\{NWTOC,CTONW,CHECKUPD,STAGE}.BAT` v1.x files** — those are abandoned drafts riddled with NT-only constructs (`FOR /F`, `SET /A`, `CALL :label`, `2>NUL` stderr redirection, tilde path modifiers). The scripts that actually run on stations live under `COMMON\ProdSW\` and were deliberately cleaned of those constructs (except for the `/Y` issue above). +- **F3 (MED):** `C:\Shares\test\TS-21\ProdSW` is a stray **file** (a misplaced `7BMAIN4`-type EXE), not the directory rsync expects — this makes the 15-minute AD2↔NAS sync report `ERRORS` on every single run (`exit 3`, "Not a directory"), masking any genuine new failure. Fix: remove/relocate the file, harden the push loop to skip non-directory `ProdSW` paths. +- **F4 (MED):** Server-side datasheet generation (`spec-reader.js`) reads specs from `testdatadb\specdata\`, which is a **frozen 2026-03-27 snapshot** — not the live engineering masters. Any spec limit changed after that date is not reflected in generated datasheets even though the outbound data pipeline itself is healthy. +- **F5 (MED, security):** Plaintext credentials found hard-coded in scripts on AD2 — rsync daemon password in `Sync-FromNAS-rsync.ps1`, NAS root SSH password in the dormant `Sync-FromNAS.ps1`, and the Postgres `testdatadb_app` password in `testdatadb\database\db.js`. Flagged for rotation + proper vaulting; not yet remediated as of 2026-07-04. +- **Stale-assumption corrections established by this audit** (do not carry forward the old versions): datastore is **PostgreSQL 18**, not SQLite (the SQLite file is a 4.4 GB archive from the 2026-04-03 migration cutover); the scheduled sync task runs **`Sync-FromNAS-rsync.ps1`**, not the dormant per-file SCP script; web delivery is a live **HTTP API uploader** (`upload-to-api.js`, 472,290 records flagged as of 2026-07-01), not the dead `For_Web`/ASP.NET path (dead since 2026-05-11); `CTONWTXT.BAT` IS actively invoked (called from `CTONW.BAT` line 30), contradicting an earlier assumption that it was a gap. +- Full report: `clients/dataforth/docs/audits/2026-07-01-test-data-chain-audit-AD2.md`. Recommendations in that report are proposals only — nothing has been applied (the audit was strictly read-only). + +### Hardware / Endpoint — Aging Fleet +- **DFORTH-Ship recurring TDR BSOD (`0x116 VIDEO_TDR_FAILURE`), diagnosed 2026-06-25:** integrated Intel HD Graphics 4600 on driver 20.19.15.5126 (Intel's final driver for that part, dated 2020-01-20) hitting the GPU-reset timeout on an 11.5-year-old HP EliteDesk 800 G1 USDT (BIOS 2014-12-10). Five minidumps span 2025-11-03 through 2026-06-24 with an **accelerating cadence** — treat as a degrading-hardware trend, not a one-off. Mitigation applied: Edge hardware acceleration disabled via machine policy (`HKLM\SOFTWARE\Policies\Microsoft\Edge\HardwareAccelerationModeEnabled = 0`). No durable fix is possible for integrated graphics (nothing to reseat/replace) — **PC replacement is the real fix**; thermal cleaning of the USDT chassis is a secondary mitigation worth doing regardless. Do not confuse with near-twin host **DForth-Shipp** — verify the exact agent ID before acting. + ### Security - **C2 IP blocks are iptables only** — do not survive UDM reboot. Must add to permanent UniFi block list via UI. C2 IPs: 80.76.49.18, 45.88.91.99 (AS399486 Virtuo, Montreal). - **AD1 disk 90% full** — C:\Engineering = 787 GB of 1023 GB. Risk of replication failures. - **Windows Firewall disabled on AD2** (all profiles) — known risk, not yet remediated. - **3 Windows 7 machines on network** (LABELPC, LABELPC2, D2-RCVG-003) — EOL, unpatched. -- **AD1/AD2 on Windows Server 2016** — end of mainstream support. Plan upgrade. +- **AD1/AD2 on Windows Server 2016 / 2019 respectively** — approaching/at end of mainstream support. Plan upgrade. - **Entra ID P2 not licensed** — IdentityRiskyUser risk check returns 403 even with scope consented. Would need P2 upgrade to enable Identity Protection. - **IdentityRiskyUser.Read.All scope:** Consented to Security Investigator app but unusable (no P2 license). +- **Plaintext credentials in Dataforth test-data-chain scripts (F5, 2026-07-01):** rsync daemon, NAS root, and Postgres app passwords are hard-coded in scripts under `C:\Shares\test\scripts\` and `testdatadb\database\db.js`, world-readable via the `test` SMB share. Rotation + vaulting recommended, not yet done. +- **Legacy appliance with no prior credential (MYDATA TPSys, 2026-07-04):** an entire production SMT-line controller existed with no vault or wiki entry until physically discovered. Root password reset via LILO recovery; now vaulted. Worth a broader sweep for other undocumented devices on VLAN 2 "mydata" given 3 unnamed industrial MACs were already known but unresolved as of 2026-06-01. ### Syncro Asset Management - **Fleet-wide Syncro agent break ~2025-10-06:** ~half of Dataforth machines stopped reporting to Syncro on or around that date while remaining online in ScreenConnect. Do NOT auto-remove machines frozen at that date without cross-checking ScreenConnect. Root cause unknown — needs investigation. @@ -437,7 +487,15 @@ Syncro asset IDs: 23845, 149614, 9708445, 9357407, 9276901, 9212922, 9078651, 88 ## Active Work -As of 2026-06-23 (no open Syncro tickets): +As of 2026-07-04 (0 open Syncro tickets per live pull): + +- **DOS Test Station Data Pipeline (Syncro #32489, active):** Root cause confirmed 2026-07-01 via a read-only ground-truth audit run through a headless Claude spawned on AD2's GuruRMM agent (new capability — see [RMM-Spawned Claude](#rmm-spawned-claude-on-ad2)). F1 (NWTOC v5.0 never copies master `.DAT` specs to stations) is confirmed; F2 (`COPY /Y` may not be valid on true MS-DOS 6.22) needs a station-side check before scoping the fix. **Next steps:** (1) confirm station DOS version (`VER` + `COPY /Y NUL C:\TEST.TXT` on a station), (2) draft a DOS-6.22-safe `NWTOC v5.1` that adds a one-way pull of master `.DAT`s (plain `COPY`, no `/Y` if 6.22 confirmed) without reintroducing the cyclic-overwrite problem v5.0 was avoiding, (3) Grok-review the new script before it touches a station, (4) update ticket #32489 with the confirmed root cause and plan. Secondary cleanup items from the same audit (not urgent): remove the stray `TS-21\ProdSW` file (F3), feed `testdatadb\specdata\` from live engineering masters (F4), rotate/vault the plaintext creds found in scripts (F5), retire dead `For_Web` output and abandoned v1.x script drafts (F6/F7/F8). + +- **MYDATA TPSys SMT controller (new, discovered 2026-07-04):** Root password reset via LILO recovery and **vaulted 2026-07-04** at `clients/dataforth/mydata-smt.sops.yaml` (host/VLAN/OS/accounts/recovery-method documented; decrypt-verified). **Outstanding:** (1) confirm the machine's exact IP on 192.168.1.x and reconcile against the known mydata VLAN member list; (2) verify the `tpsys` wheel-group + scoped `NOPASSWD` sudoers change actually landed (`id tpsys`, `sudo -l` as tpsys); (3) get the exact TPSys app-launch command from Howard/Mike to finalize the sudoers scope; (4) confirm the controller booted cleanly into TPSys after the forced reboot (it is a live production SMT line); (5) decide and stand up agentless monitoring (ICMP/TCP probe or SSH heartbeat from D2TESTNAS or the RMM server — inter-VLAN routing to mydata is open) since a GuruRMM agent is impossible on this OS; formalize via `/feature-request` if Mike wants legacy/appliance Linux monitoring as a standing GuruRMM capability. + +- **DFORTH-Ship BSOD (ongoing monitoring):** Edge hardware-acceleration mitigation applied 2026-06-25; needs on-site Edge restart/reboot to take effect, verify at `edge://policy`. Monitor for recurrence — if it bugchecks again, pull and analyze the four older dump signatures to confirm whether it is drifting toward a hard hardware fault. Schedule thermal cleaning of the USDT chassis. Recommend/plan replacement of the 11.5-year-old EliteDesk 800 G1 USDT shipping station as the durable fix. + +- **UDM inbound SIP DNAT (recurring risk, unresolved):** Confirmed again 2026-06-23 that the SIP 5060 DNAT can be flushed by a UniFi controller provision mid-uptime, not only at reboot. Coord to-do `45572ee1` tracks the durable fix (persistent UI port-forward rule or a cron/watcher re-running the idempotent on_boot.d script) — needs a maintenance window. Still SIP-only; never forward the RTP range. - **Shares & Permissions project (Phase 1 — BLOCKING, pending client input):** Phase 0 (discovery) completed 2026-06-10 — read-only ACL audit confirmed all 8 business shares open to all employees; Domain Users has FullControl on 4 shares. Discovery email to Dan Center drafted (`clients/dataforth/docs/projects/shares-permissions/discovery-email-draft.md`); **not yet sent — recipients/sender not locked** (Dan Center primary; CC Kevin Wackerly?; Mike or Howard sending?). Phase 1 blocked on client responses: department list, access matrix, sensitive-data rules, staff rosters. A **Phase 2 target-state strawman was drafted 2026-06-22** (`target-structure-draft-2026-06-22.md` + client-facing `Dataforth-Shared-Drives-Plan.html`) from the existing layout — see [Shares ACL State](#shares-acl-state--all-open-to-all-staff); it still needs the Phase 1 client matrix to finalize. Next-step options: polish the client HTML, finalize + send the discovery email to unblock Phase 1, or refine the internal strawman. Full roadmap: `clients/dataforth/docs/projects/shares-permissions/roadmap.md`. @@ -449,12 +507,13 @@ As of 2026-06-23 (no open Syncro tickets): - **AOI XP backup + isolation (ongoing):** AOI optical-inspection XP PC on VLAN 2 (mydata/SMT) @ 192.168.1.175; locked-down SMB1 share `aoibackup` on D2TESTNAS (XP-only, user `admin`). Other NAS shares now deny the XP. **Optional EOL hardening pending:** block XP → company LAN (except NAS 192.168.0.9) + Internet on the UDM, scoped to .175. Todo `37543f7f`. -- **AD2 Claude capability updates (parked):** AD2 runs its own Claude from `C:\ClaudeTools` on the `ad2` branch. Needs: (a) syncro + coord commands, (b) DF wiki read-write, (c) Dataforth client data access. Python 3.12.8 and identity.json installed 2026-06-17. Coord API unreachable from Dataforth LAN — comms via git sync only. +- **AD2 Claude capability updates (parked):** AD2 runs its own Claude from `C:\ClaudeTools` on the `ad2` branch. Needs: (a) syncro + coord commands, (b) DF wiki read-write, (c) Dataforth client data access. Python 3.12.8 and identity.json installed 2026-06-17. Coord API unreachable from Dataforth LAN — comms via git sync only, though the 2026-07-01 RMM-spawn pattern now offers a read-only side channel for investigation. - **Power Monitor SPA demo (parked):** Georg Haubner developed a vanilla-JS power-meter SPA (AI-built, `clients/dataforth/ExternalCodeReview.zip`). ACG designed a gateway architecture for a gated demo at `PWM.dataforth.com` (inbound tunnel, no meter publicly exposed, magic-link auth). Spec at `clients/dataforth/power-monitor-demo/GATEWAY-SPEC.md`. Parked pending Mike↔Georg conversation. - **Test Datasheet Pipeline:** - - Production pipeline healthy. 469K records, DSCA33/45 recovery complete (1,452 new certs published 2026-06-18 via Hoffman API). Daily task runs 02:30 AM. + - Production pipeline healthy — outbound (station → NAS → AD2 → Postgres → web) confirmed current as of 2026-07-01 (import as recent as 13:41 UTC same day). 475K+ records, DSCA33/45 recovery complete (1,452 new certs published 2026-06-18 via Hoffman API). + - Inbound spec/software distribution to stations is broken — see DOS Test Station Data Pipeline pattern above and Syncro #32489. - Email notifications deployed (Graph API via `sysadmin@dataforth.com`). - 8B/5B/SCM render gap — parked with AD2 (see above). - 2 niche DSCA models (DSCA33-1948, DSCA45-1746) and their 8B equivalents have no Hoffman original — no template, cannot auto-publish. @@ -470,8 +529,6 @@ As of 2026-06-23 (no open Syncro tickets): - **C2 IP blocks need permanence:** Iptables rules on UDM (80.76.49.18, 45.88.91.99) need to be added to permanent UniFi UI block list. -- **UDM inbound SIP port-forward:** Recommended to add matching rule in UniFi UI (current on_boot.d script covers reboots; UI rule is belt-and-suspenders). - --- ## History Highlights @@ -511,6 +568,10 @@ As of 2026-06-23 (no open Syncro tickets): | 2026-06-17 | AD2 identity.json + Python 3.12.8 installed. `CLAUDE.dataforth.md` created for AD2 context file (relocated from in-line `.claude/CLAUDE.md` edits to maintain clean fork). | | 2026-06-18 | **DSCA33/45 certs recovered via Hoffman API** — 56 model templates mined, 1,452 new DSCA33/45 certs published on AD2 (0 overwrites). Root-caused `parseRawData` bug affecting 8B/5B/SCM families. 136 8B/5B/SCM templates mined from Hoffman and handed to AD2 for wiring. TestDataDB UI redesigned and deployed on AD2 (cert-fit, publish chips, push toasts, full-screen inspector). AD2 SSH PMTU blackhole diagnosed (GURU-5070 adapter MTU 1500 vs tunnel ~1424) and fixed (MTU 1400). Syncro #32441. | | 2026-06-22 | **Shares & Permissions Phase 2 target-state strawman drafted** — proposed `Company\Departments\…Restricted\…Company-Wide\…Users\…Archive\` tree with `SG--` groups, current→target migration map, and Option-A (keep drive letters) rollout, all inferred from the existing layout. Internal draft + client-facing HTML render. Phase 1 client input still gates sign-off. | +| 2026-06-23 | **PBX no-inbound-calls emergency fixed.** UDM SIP 5060 DNAT was completely absent — flushed by a UniFi controller provision (not a reboot), confirming the on_boot.d re-apply script's coverage gap is broader than previously understood. Re-ran `/data/on_boot.d/30-freepbx-sip-forward.sh`, verified DNAT + forward-accept restored, inbound confirmed working end-to-end. Vault `pbx.sops.yaml` password corrected (was backslash-corrupted from shell-escaping leaking into storage). Syncro #32450, 1.5 hr emergency (×1.5 rate) remote, invoiced, block debited 31.5 → 30.0 hrs. Durable-fix coord to-do `45572ee1` filed. | +| 2026-06-24–25 | **DFORTH-Ship recurring BSOD diagnosed.** Stop code `0x116 VIDEO_TDR_FAILURE` on integrated Intel HD Graphics 4600 (final 2020 driver) — an 11.5-year-old HP EliteDesk 800 G1 USDT with an accelerating crash cadence (5 dumps since 2025-11-03). Mitigated by disabling Edge hardware acceleration via machine policy; PC replacement recommended as the durable fix; thermal cleaning flagged as a secondary measure. | +| 2026-07-01 | **Test-data-chain ground-truth audit** via a headless Claude spawned through AD2's GuruRMM agent (new capability, bypassing AD2's coord-API isolation for read-only investigation). Confirmed root cause of Syncro #32489: deployed `NWTOC.BAT` v5.0 never copies master spec `.DAT` files to stations (removed by design in the v5.0 changelog). New HIGH finding: `NWTOC.BAT`/`CTONWTXT.BAT` use `COPY /Y`, not a valid MS-DOS 6.22 switch — pending station-side DOS-version confirmation before scoping the fix (independently confirmed by Grok). Corrected several stale assumptions: datastore is PostgreSQL 18 (475,553 test_records), the sync task runs `Sync-FromNAS-rsync.ps1`, web delivery is a live HTTP API uploader (472,290 records flagged). Also found: a stray file breaking the AD2↔NAS rsync push on every run (masking real errors), a frozen 2026-03-27 server-side spec snapshot, and plaintext credentials hard-coded in three scripts (flagged for rotation, not yet fixed). | +| 2026-07-04 | MYDATA TPSys SMT controller (myserver, FC3/VLAN2) discovered + root recovered via LILO single-user; vaulted; RMM agent ruled out (legacy glibc/kernel/no-systemd). | --- diff --git a/wiki/index.md b/wiki/index.md index b3d0b913..fb045a1f 100644 --- a/wiki/index.md +++ b/wiki/index.md @@ -1,6 +1,6 @@ # Wiki Index -Last updated: 2026-07-03 +Last updated: 2026-07-04 Compiled by: HOWARD-HOME/claude-main This wiki is LLM-maintained. Do not edit articles manually — run `/wiki-compile` to update. @@ -19,7 +19,7 @@ Run `/wiki-lint` to check for stale entries and broken backlinks. | Article | Summary | Last Compiled | |---|---|---| | [Cascades of Tucson](clients/cascades-tucson.md) | Prepaid block $175/hr, **37.5 hrs remaining** (live 2026-07-01); senior living; active domain migration + HIPAA caregiver-lockdown project (GPOs deployed; Entra Hybrid Join + CA allow-list + ALIS SSO model proven); single DC (CS-SERVER) on aging R610 -- RAID **live-verified HEALTHY 2026-06-24** (the 6/15 "degraded" self-recovered; both mirrors Ok, 1:0:4 = global hot spare; consumer 320GB drives + lost-PSU-redundancy are planned follow-ups, NOT an emergency); cloud backup verified running; **Planned power outage 2026-06-23** clean self-shutdown executed + verified (bring-up ~09:00, John onsite); **Voice VLAN 30 migration COMPLETE 2026-06-19** (~38 devices: 29 Poly + 8 AudioCodes + desktop; awaiting Vertical to set Poly 5GHz-only); **UniFi RF optimized 2026-06-19** (77 U7-Pro APs/~587 clients: 2.4GHz power->Medium on 47 radios + 5GHz clean-DFS 40MHz channel plan -> 5GHz retry halved; 6GHz blocked by WPA3 on PPSK SSID); Syncro 0 open tickets (live EOD 2026-06-25), device-readiness audit done (5 PCs on Win Home need Home->Pro before join); **Alma Montt offboarded 2026-06-25** (Tenant Admin SP left holding a standing PAA role -- removal pending Mike); **CARF Technology & System Plan** deliverable in progress for Ashley Jensen; **endpoint security migration started 2026-06-25** (Datto EDR/AV replacing Bitdefender; 34 agents enrolled); **CS-SERVER: all Datto software removed 2026-06-26**, and the CS-SERVER "SMB error 67" proved to be an RMM-test artifact -- server is healthy, Karen share access verified interactively; **caregiver phone login LIVE 2026-07-01: roster reconciled to 35 (8 offboarded incl. Lassey dup, 4 new hires), two CA blockers fixed (MFA-exclude bug + compliance-block disabled), interim posture = desktops+phones on-network only; ALIS Email=UPN match + 3 new-hire ALIS records pending**; **CSC ENT->VLAN 20 migration (live 2026-07-01): 22 machines on VLAN 20 (only CS-SERVER + ~6 stragglers left on old LAN); printer shares lag at 4/15 repointed; P&P GPO still pilot-scoped**; remaining-work plan: docs/REMAINING-WORK-PLAN.md | 2026-07-01 | -| [Dataforth Corporation](clients/dataforth.md) | Prepaid block ~$2,099/mo, **31.5 hrs remaining** (live 2026-06-23); signal-conditioning manufacturer; 64 DOS test stations; 2025 ransomware recovery + incomplete file restore (migration-gap audit); 2026-03 phishing + MFA rollout; test-datasheet pipeline (DSCA cert publish via Hoffman API + testdatadb UI on AD2); mail stack INKY->Mailprotector CloudFilter->EXO; FreePBX 17 outage fixed 2026-06-08/09 (qualify_frequency=0; no RTP-forward); shares-ACL project (all open to staff; Phase 2 target-state strawman drafted 2026-06-22); Syncro asset reconciliation 2026-06-02; GuruRMM fleet ~45; Bitdefender phase-off | 2026-06-23 | +| [Dataforth Corporation](clients/dataforth.md) | Prepaid block ~$2,099/mo, **30.0 hrs remaining** (live 2026-07-04, 0 open tickets); signal-conditioning manufacturer; 64 DOS test stations; **new 2026-07-04: undocumented MYDATA TPSys SMT line controller (`myserver`, Fedora Core 3, VLAN2) discovered — root recovered via LILO single-user + vaulted; RMM agent ruled out (legacy glibc/kernel/no-systemd)**; test-data-chain audit 2026-07-01 (NWTOC v5.0 spec-copy gap = Syncro #32489); PBX inbound-SIP DNAT emergency fixed 2026-06-23; DFORTH-Ship 0x116 BSOD (aging G1 USDT); 2025 ransomware recovery + incomplete file restore; 2026-03 phishing + MFA; test-datasheet pipeline (Hoffman API + testdatadb on AD2); mail stack INKY->Mailprotector->EXO; shares-ACL project (all open to staff; Phase 2 strawman 2026-06-22); GuruRMM fleet ~45; Bitdefender phase-off | 2026-07-04 | | [Instrumental Music Center](clients/instrumental-music-center.md) | Prepaid block $175/hr, 12.5 hrs remaining; music retail/repair; AIMsi POS on SQL Server 2019; phantom DC causing slow logons; GuruRMM enrolled (IMC1) | 2026-05-24 | | [Jimmy Company](clients/jimmy.md) | Break-fix, $150/hr; single aging workstation BLASTER2 (Win10 22H2 EOL, i5-3470/3.8GB — replace); backups the recurring theme (QuickBooks data); onboarded to GuruRMM 2026-06-19 (RDP NLA + Kaseya removal + cleanup); MSP360 local backup drive full, 90-day retention set, space reclaim pending in console (cloud B2 healthy) | 2026-06-19 | | [Valley Wide Plastering](clients/valleywide.md) | Prepaid block, 15.5 hrs remaining; plastering/stucco contractor; HP DL360 Gen10 + XenServer; VWP-FILES (G:) on Hyper-V — SMB1 enabled for the legacy XP Orders VM (V-XP); VB6 app modernization project; RDWeb brute-force incident; 11 Yealink phones pending | 2026-06-23 |