claudetools/clients/dataforth/session-logs/2026-07/2026-07-05-mike-dos-spec-fix-telemetry-udm.md

User

• User: Mike Swanson (mike)
• Machine: GURU-5070
• Role: admin

Session Summary

Resumed Syncro #32489 (Dataforth DOS test stations not pulling updated spec files on restart) and drove it to a verified fleet-wide fix, then built station telemetry, then handled a live UDM incident. Starting from the 2026-07-01 AD2 audit, confirmed the root cause: the deployed on-boot updater NWTOC.BAT v5.0 (2026-03-16) stopped copying master spec .DAT files (F1) and used COPY /Y, which is not a valid MS-DOS 6.22 switch (F2). Mike confirmed all stations are genuine 6.2/6.22 (no 7.x), which meant v5.0's /Y had been failing on all three of its copy steps fleet-wide since March — stations received no BAT/EXE/spec updates at all. Built NWTOC v5.1 (restores a one-way *.DAT pull from the read-only engineering master into C:\ATE\<type>DATA, plain COPY, ATTRIB -R guard), plus SPECCHK.BAT and a .gitattributes pinning *.BAT to CRLF. Grok reviewed the DOS-6.22 correctness.

Deployed the whole toolchain to the NAS COMMON\ProdSW by dispatching read-only + write PowerShell to AD2's GuruRMM agent, which pushes to the NAS via its own rsync daemon creds (never exposed). Validated NWTOC v5.1 on TS-3L (specs pulled, clean boot to menux), then promoted it to fleet master (v5.0 backed up to NWTOC50.BAK). Because the broken v5.0 cannot bootstrap its own replacement (broken /Y), the rollout is a controlled per-station opt-in: each station gets one manual plain-COPY seed, then self-maintains. Also shipped CTONWTXT v2.4 (drops /Y).

Built Tier-A station telemetry to collect each station's CONFIG.SYS/AUTOEXEC/SPECVER/MEM to T:\STATUS. The first attempt (CTONW v5.1) used a network MD guarded by the unreliable IF NOT EXIST ...\NUL test with no trailing backslash on the COPY targets — on the DF stations this created files-named-after-machines instead of folders and triggered a T: write fault critical-error that HUNG unattended boots on production stations. Reverted CTONW to v5.0 immediately (self-heals since NWTOC pulls the reverted copy before CTONW runs), root-caused it, and rebuilt as CTONW v5.2 using Mike's flat-file approach (%MACHINE%.CFG/.AUT/.SPV/.MEM, explicit filename targets, no MD). Promoted v5.2; all 14 stations collected cleanly. Confirmed via the collected SPECVER stamps that all 14 active DOS stations carry the current 5BMAIN.DAT (83,200, 06-26-26) — #32489 fixed fleet-wide (Mike noted 14 is the entire remaining DOS fleet; the rest are retired). The CONFIG.SYS collection revealed 6 hardware-specific variants (SETADDR.SYS 278 vs 378, USBASPI, LPT2), so a single master CONFIG.SYS was ruled out. Added SYSTAT.BAT (plain, paused status page showing batch versions + spec dates before menux) and CTONW v5.3 to wire it in.

Late in the session Mike reported the Dataforth UDM showing 0 connected clients. Diagnosed it via SSH (root key auth from GURU-5070): data plane fine, mongo ace DB up, no eMMC errors — but the UniFi Network app's RabbitMQ/websocket messaging was broken (transformer has been closed, WebSocket session has been closed) under memory pressure. Restarted unifi.service and clients repopulated. Root cause was memory starvation from UniFi Protect + Access + Talk being installed on a 4 GB UDM (Talk 1.1 GB, all idle/unused — Dataforth uses FreePBX). Verified none were in use, Mike uninstalled all three via the UI, and the UDM went unreachable. I incorrectly assumed it had rebooted/gone down and asked Mike to eyeball hardware; Mike corrected that I should verify via RMM. Verified through the D2TESTNAS RMM agent that the DF LAN is up (machines checking in) but the UDM's management plane (SSH/ICMP) is still down while it forwards — consistent with finishing its post-uninstall reboot. UDM verification is still pending.

Key Decisions

• Used plain COPY (not /Y, not the version-agnostic DEL+COPY) in NWTOC v5.1 per Mike's choice — the fleet is uniformly 6.22 today; the 7.x-upgrade landmine is documented in the file header for a future migration runbook.
• Deployed all batch files via AD2's RMM agent → its own rsync-daemon push to the NAS, rather than SSH from GURU-5070, so the rsync secret never left AD2 and the push used the proven Sync-FromNAS mechanism.
• Staged each new/changed batch file under a distinct name (NWTOC51.BAT, CTONW52.BAT, CTONW53.BAT) for single-station validation before promoting over the live filename — so the fleet copy stays safe until proven.
• After the boot-hang incident, adopted Mike's flat-file telemetry (machine-named files in an existing T:\STATUS, no network MD, explicit filename COPY targets) — eliminates the entire directory-creation failure class.
• Restarted only unifi.service on the UDM (not a full reboot) to fix the 0-clients — the data plane stays up; a UDM reboot would drop the whole site.
• Ruled out a single master CONFIG.SYS after collection showed hardware-specific per-station differences (SETADDR I/O addresses); the ANSI/standardization path must be additive, not a wholesale replace.

Problems Encountered

• CRLF false-positive: the Write tool emitted LF-only .BAT files, and my grep -c $'\r$' check falsely reported CRLF. Staged an LF-only NWTOC51.BAT to the NAS; Mike caught it at the station. Fixed by converting with sed 's/$/\r/' and verifying with tr -cd '\r' | wc -c (must equal LF count) and xxd. Logged to errorlog (--friction).
• CTONW v5.1 boot hang (production): network MD + IF NOT EXIST ...\NUL (unreliable on SMB) + COPY without trailing backslash → files instead of folders and a T: write fault Abort/Retry/Fail that hung unattended boots. Reverted to v5.0; rebuilt as flat-file v5.2. Logged to errorlog.
• UDM verification — wrong assumption: when the UDM went unreachable after the app uninstall, I asserted it had rebooted/died and asked Mike (who is remote, not on-site) to check hardware. Correct approach was to verify via the D2TESTNAS RMM agent, which showed the DF LAN up and the UDM forwarding but management-plane down. Corrected mid-thread.
• Stale RMM agent id: AD2 re-enrolled during the session (id cfa93bb6… → 4b9fe588…), so the CTONW v5.3/SYSTAT promotion dispatched to the old id sat pending and never ran. Needs redoing against the new id.
• RMM API auth/truncation blips: intermittent 401s and truncated large responses to 172.16.3.30 over the DF→ACG link; re-auth cleared them.

Configuration Changes

Created (repo, projects/dataforth-dos/, uncommitted in submodule as of save):

• batch-files/NWTOC.v5.1-draft.BAT, CTONW.v5.1-draft.BAT (withdrawn), CTONW.v5.2-draft.BAT, CTONW.v5.3-draft.BAT, CTONWTXT.v2.4-draft.BAT, STATUS.v1-draft.BAT (ANSI), SYSTAT.v1-draft.BAT (plain paused), SPECCHK.v1-draft.BAT
• .gitattributes (*.BAT/*.CMD → text eol=crlf)
• docs/TIER-A-STATION-TELEMETRY.md

Deployed to NAS COMMON\ProdSW (via AD2 RMM → rsync), all CRLF + sha-verified:

• NWTOC.BAT = v5.1 (5806 B, sha fc2f683a); backup NWTOC50.BAK (v5.0, 972 B)
• CTONW.BAT = v5.2 (2177 B, sha 8d2f3acc) live; CTONW52.BAT (v5.2), CTONW53.BAT (v5.3, sha 99cecc49, NOT yet promoted — dispatch went to stale AD2 id); backup CTONW50.BAK (v5.0, 1504 B)
• CTONWTXT.BAT = v2.4 (490 B, sha 46bc8d71); backup CTONWTXT.BAK (v2.3, 392 B)
• STATUS.BAT (v1.1 ANSI, 1976 B, sha e8b2c6f4) — superseded by SYSTAT, left in place
• SYSTAT.BAT (v1.1 plain paused, 925 B, sha e85e7c…)

UDM 192.168.0.254: restarted unifi.service (Network app; PID 25420 → 730335). Mike uninstalled UniFi Protect 7.1.83, Access 4.2.29, Talk 5.2.7 (+ freeswitch) via UI.

errorlog.md: 2 entries (CRLF friction; CTONW boot-hang). No repo commits pushed this session (submodule work still uncommitted).

Credentials & Secrets

No new credentials created or discovered. Used (read-only) from vault:

• infrastructure/gururmm-server.sops.yaml — RMM API admin login
• clients/dataforth/udm.sops.yaml — UDM 192.168.0.254; SSH root key auth from GURU-5070 works (BatchMode succeeded); admin azcomputerguru (2FA push on web); MongoDB 127.0.0.1:27117/ace
• AD2 rsync-daemon password read at runtime from AD2's own Sync-FromNAS-rsync.ps1 ($RSYNC_PASSWORD), never returned off-box; NAS rsync rsync://rsync@192.168.0.9/test

Infrastructure & Servers

• AD2 192.168.0.6 — Dataforth DC, GuruRMM agent re-enrolled: new id 4b9fe588-2a86-4855-912e-d90cf5f5e3e4 (old cfa93bb6-… now stale), site "D2". cwRsync at C:\ProgramData\chocolatey\bin\rsync.exe.
• D2TESTNAS 192.168.0.9 — DF Linux storage/NAS + UDM jump host. RMM agent hostname ix.azcomputerguru.com, id 4ad2e426-b03f-4c5d-817c-c8c675ba73a0, site "DF Server Storage", os linux. rsync daemon module test → /data/test.
• UDM 192.168.0.254 — Dataforth gateway/firewall, UbiOS, 4 GB RAM. Was running Network + Protect + Access + Talk; unifi.service = Network app (Java, -Xmx512M); mongo ace on 127.0.0.1:27117. FreePBX SIP forward in /data/on_boot.d/30-freepbx-sip-forward.sh (→ 192.168.100.2). Reached from GURU-5070 via Tailscale (D2TESTNAS subnet router) — that route dropped when the UDM management went down.
• GuruRMM API http://172.16.3.30:3001. 14 active DOS stations: TS-1L/1R, TS-3L/3R, TS-4L/4R, TS-8L/8R, TS-10L/10R, TS-11L/11R, TS-12, TS-27 (no RMM agents — DOS).

Commands & Outputs

• Deploy pattern: curl POST $RMM/api/agents/<id>/command (jq-built payload) → poll /api/commands/<id>; PowerShell decodes gzip+base64 batch content, sha-verifies, rsync -v -t push to rsync://rsync@192.168.0.9/test/COMMON/ProdSW/.
• Line-ending truth check: tr -cd '\r' < f | wc -c must equal tr -cd '\n' < f | wc -c; head -c 60 f | xxd should show 0d 0a. NOT grep -c $'\r$' (false-positived).
• SPECVER verification: all 14 .SPV files show 5BMAIN DAT 83,200 06-26-26 11:43a.
• UDM diagnosis: server.log full of transformer has been closed / WebSocket session has been closed; free -m ~1 GB swap used; systemctl restart unifi → clients repopulated, mongo ace db.user.count()=300.
• UDM apps: dpkg -l → unifi-protect 7.1.83, unifi-access 4.2.29, unifi-talk 5.2.7, freeswitch-unifi-talk; footprint Talk 1.1 G / Protect 242 M / Access 19 M; Protect 0 recorded video files.
• UDM post-uninstall: from D2TESTNAS RMM shell, ping 192.168.0.254 = DOWN, SSH timed out — but DF machines checking into RMM (data plane up).

Pending / Incomplete Tasks

• UDM: re-probe via D2TESTNAS RMM until SSH/ICMP returns; then confirm Protect/Access/Talk gone and memory recovered, and Network app came up clean. (Was mid post-uninstall reboot: forwarding up, management down.)
• SYSTAT/CTONW v5.3: re-dispatch the promote to the NEW AD2 id 4b9fe588… (old dispatch stale); confirm SYSTAT paused page appears before menux.
• Commit submodule work: the 8 batch drafts + .gitattributes + docs/ in projects/dataforth-dos/ are uncommitted (plain sync skips submodule internals). Commit inside the submodule, then --with-submodules sync to bump the pointer.
• #32489 ticket: John reply drafted (customer-visible, held per Mike until all changes done); decide status (Resolved vs keep open) and billing (Mike chose to bill — needs minutes + labor type; check Dataforth prepaid block).
• CONFIG.SYS/ANSI: additive DEVICE=C:\DOS\ANSI.SYS per station (preserving hardware lines) if pursuing ANSI, not a master file.
• UDM memory (root cause): removing Protect/Access/Talk is the durable fix; verify RAM headroom post-removal so the 0-clients doesn't recur.
• Clean stray extension-less files in T:\STATUS (from the v5.1 incident).

Reference Information

• Syncro #32489 (id 113201089), Dataforth Corp (customer 578095), contact John Lehman (2851723), was "Customer Reply", past due 2026-07-02.
• Prior audit: clients/dataforth/docs/audits/2026-07-01-test-data-chain-audit-AD2.md; project: projects/dataforth-dos/.
• Batch source of truth: projects/dataforth-dos/batch-files/. Tier-A design: projects/dataforth-dos/docs/TIER-A-STATION-TELEMETRY.md.
• Boot flow: AUTOEXEC → STARTNET → NWTOC (download) → CTONW (upload) → menux. NWTOC pulls COMMON\ProdSW\*.BAT → C:\BAT each boot (self-update path; broken v5.0 /Y cannot bootstrap v5.1).
• SHAs above are file content hashes (sha256) used for deploy verification, not git commits.

Update — UDM incident resolved + SYSTAT wired

• UDM incident CLOSED. VPN restored the 192.168.0.x route; direct SSH confirmed: unifi-protect/access/talk + freeswitch packages and services all gone, memory recovered (available 998 MB → 1371 MB, used down ~380 MB), unifi.service healthy (RSS ~365 MB, was ~450 MB), ace DB 299 clients. UDM uptime 16 days — it never rebooted; the earlier "unreachable" stretch was a routing gap, not the UDM going down. Root cause (memory starvation from three unneeded apps on a 4 GB UDM) eliminated → recurrence unlikely.
• Correction / topology: D2TESTNAS (RMM agent ix.azcomputerguru.com) is on the 172.16.x segment (default gw 172.16.0.1) and cannot reach 192.168.0.x (no ARP for the UDM, can't ping AD2 either) — so it was the wrong vantage point; its "UDM down" reads meant "D2TESTNAS can't route there." The on-segment vantage point is AD2 (192.168.0.6), or a direct VPN/Tailscale route. Twice this session, verifying beat assuming (the boot-hang, and this).
• SYSTAT wired: CTONW v5.3 promoted live via AD2's new RMM id 4b9fe588 (old cfa93bb6 stale after re-enroll). CTONW.BAT = v5.3 (2121 B); paused SYSTAT status page now runs before menux fleet-wide. Rollback CTONW52.BAT (v5.2).
• Still open: #32489 John reply (drafted, held) + billing (needs minutes + labor type) + ticket status.