azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sync: auto-sync from GURU-5070 at 2026-07-05 16:35:57

Browse Source 
Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-07-05 16:35:57
This commit is contained in:
Mike Swanson
2026-07-05 16:36:27 -07:00
parent bee140c020
commit 5e682c9293
1 changed files with 90 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

90
clients/dataforth/session-logs/2026-07/2026-07-05-mike-dos-spec-fix-telemetry-udm.md Normal file
View File

@@ -0,0 +1,90 @@
## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin
## Session Summary
Resumed Syncro #32489 (Dataforth DOS test stations not pulling updated spec files on restart) and drove it to a verified fleet-wide fix, then built station telemetry, then handled a live UDM incident. Starting from the 2026-07-01 AD2 audit, confirmed the root cause: the deployed on-boot updater `NWTOC.BAT` v5.0 (2026-03-16) stopped copying master spec `.DAT` files (F1) and used `COPY /Y`, which is not a valid MS-DOS 6.22 switch (F2). Mike confirmed all stations are genuine 6.2/6.22 (no 7.x), which meant v5.0's `/Y` had been failing on all three of its copy steps fleet-wide since March — stations received no BAT/EXE/spec updates at all. Built `NWTOC v5.1` (restores a one-way `*.DAT` pull from the read-only engineering master into `C:\ATE\<type>DATA`, plain `COPY`, `ATTRIB -R` guard), plus `SPECCHK.BAT` and a `.gitattributes` pinning `*.BAT` to CRLF. Grok reviewed the DOS-6.22 correctness.
Deployed the whole toolchain to the NAS `COMMON\ProdSW` by dispatching read-only + write PowerShell to AD2's GuruRMM agent, which pushes to the NAS via its own rsync daemon creds (never exposed). Validated `NWTOC v5.1` on TS-3L (specs pulled, clean boot to menux), then promoted it to fleet master (v5.0 backed up to `NWTOC50.BAK`). Because the broken v5.0 cannot bootstrap its own replacement (broken `/Y`), the rollout is a controlled per-station opt-in: each station gets one manual plain-`COPY` seed, then self-maintains. Also shipped `CTONWTXT v2.4` (drops `/Y`).
Built Tier-A station telemetry to collect each station's `CONFIG.SYS`/`AUTOEXEC`/`SPECVER`/`MEM` to `T:\STATUS`. The first attempt (`CTONW v5.1`) used a network `MD` guarded by the unreliable `IF NOT EXIST ...\NUL` test with no trailing backslash on the `COPY` targets — on the DF stations this created files-named-after-machines instead of folders and triggered a `T: write fault` critical-error that HUNG unattended boots on production stations. Reverted `CTONW` to v5.0 immediately (self-heals since NWTOC pulls the reverted copy before CTONW runs), root-caused it, and rebuilt as `CTONW v5.2` using Mike's flat-file approach (`%MACHINE%.CFG/.AUT/.SPV/.MEM`, explicit filename targets, no `MD`). Promoted v5.2; all 14 stations collected cleanly. Confirmed via the collected `SPECVER` stamps that **all 14 active DOS stations** carry the current `5BMAIN.DAT` (83,200, 06-26-26) — #32489 fixed fleet-wide (Mike noted 14 is the entire remaining DOS fleet; the rest are retired). The CONFIG.SYS collection revealed 6 hardware-specific variants (`SETADDR.SYS 278` vs `378`, USBASPI, LPT2), so a single master CONFIG.SYS was ruled out. Added `SYSTAT.BAT` (plain, paused status page showing batch versions + spec dates before menux) and `CTONW v5.3` to wire it in.
Late in the session Mike reported the Dataforth UDM showing 0 connected clients. Diagnosed it via SSH (root key auth from GURU-5070): data plane fine, `mongo ace` DB up, no eMMC errors — but the UniFi Network app's RabbitMQ/websocket messaging was broken (`transformer has been closed`, `WebSocket session has been closed`) under memory pressure. Restarted `unifi.service` and clients repopulated. Root cause was memory starvation from **UniFi Protect + Access + Talk** being installed on a 4 GB UDM (Talk 1.1 GB, all idle/unused — Dataforth uses FreePBX). Verified none were in use, Mike uninstalled all three via the UI, and the UDM went unreachable. I incorrectly assumed it had rebooted/gone down and asked Mike to eyeball hardware; Mike corrected that I should verify via RMM. Verified through the D2TESTNAS RMM agent that the DF LAN is up (machines checking in) but the UDM's management plane (SSH/ICMP) is still down while it forwards — consistent with finishing its post-uninstall reboot. UDM verification is still pending.
## Key Decisions
- Used plain `COPY` (not `/Y`, not the version-agnostic DEL+COPY) in `NWTOC v5.1` per Mike's choice — the fleet is uniformly 6.22 today; the 7.x-upgrade landmine is documented in the file header for a future migration runbook.
- Deployed all batch files via AD2's RMM agent → its own rsync-daemon push to the NAS, rather than SSH from GURU-5070, so the rsync secret never left AD2 and the push used the proven Sync-FromNAS mechanism.
- Staged each new/changed batch file under a distinct name (`NWTOC51.BAT`, `CTONW52.BAT`, `CTONW53.BAT`) for single-station validation before promoting over the live filename — so the fleet copy stays safe until proven.
- After the boot-hang incident, adopted Mike's flat-file telemetry (machine-named files in an existing `T:\STATUS`, no network `MD`, explicit filename `COPY` targets) — eliminates the entire directory-creation failure class.
- Restarted only `unifi.service` on the UDM (not a full reboot) to fix the 0-clients — the data plane stays up; a UDM reboot would drop the whole site.
- Ruled out a single master CONFIG.SYS after collection showed hardware-specific per-station differences (SETADDR I/O addresses); the ANSI/standardization path must be additive, not a wholesale replace.
## Problems Encountered
- **CRLF false-positive:** the Write tool emitted LF-only `.BAT` files, and my `grep -c $'\r$'` check falsely reported CRLF. Staged an LF-only `NWTOC51.BAT` to the NAS; Mike caught it at the station. Fixed by converting with `sed 's/$/\r/'` and verifying with `tr -cd '\r' | wc -c` (must equal LF count) and `xxd`. Logged to errorlog (--friction).
- **CTONW v5.1 boot hang (production):** network `MD` + `IF NOT EXIST ...\NUL` (unreliable on SMB) + `COPY` without trailing backslash → files instead of folders and a `T: write fault` Abort/Retry/Fail that hung unattended boots. Reverted to v5.0; rebuilt as flat-file v5.2. Logged to errorlog.
- **UDM verification — wrong assumption:** when the UDM went unreachable after the app uninstall, I asserted it had rebooted/died and asked Mike (who is remote, not on-site) to check hardware. Correct approach was to verify via the D2TESTNAS RMM agent, which showed the DF LAN up and the UDM forwarding but management-plane down. Corrected mid-thread.
- **Stale RMM agent id:** AD2 re-enrolled during the session (id `cfa93bb6…` → `4b9fe588…`), so the `CTONW v5.3`/SYSTAT promotion dispatched to the old id sat `pending` and never ran. Needs redoing against the new id.
- **RMM API auth/truncation blips:** intermittent 401s and truncated large responses to `172.16.3.30` over the DF→ACG link; re-auth cleared them.
## Configuration Changes
Created (repo, `projects/dataforth-dos/`, uncommitted in submodule as of save):
- `batch-files/NWTOC.v5.1-draft.BAT`, `CTONW.v5.1-draft.BAT` (withdrawn), `CTONW.v5.2-draft.BAT`, `CTONW.v5.3-draft.BAT`, `CTONWTXT.v2.4-draft.BAT`, `STATUS.v1-draft.BAT` (ANSI), `SYSTAT.v1-draft.BAT` (plain paused), `SPECCHK.v1-draft.BAT`
- `.gitattributes` (`*.BAT`/`*.CMD` → `text eol=crlf`)
- `docs/TIER-A-STATION-TELEMETRY.md`
Deployed to NAS `COMMON\ProdSW` (via AD2 RMM → rsync), all CRLF + sha-verified:
- `NWTOC.BAT` = v5.1 (5806 B, sha `fc2f683a`); backup `NWTOC50.BAK` (v5.0, 972 B)
- `CTONW.BAT` = v5.2 (2177 B, sha `8d2f3acc`) live; `CTONW52.BAT` (v5.2), `CTONW53.BAT` (v5.3, sha `99cecc49`, NOT yet promoted — dispatch went to stale AD2 id); backup `CTONW50.BAK` (v5.0, 1504 B)
- `CTONWTXT.BAT` = v2.4 (490 B, sha `46bc8d71`); backup `CTONWTXT.BAK` (v2.3, 392 B)
- `STATUS.BAT` (v1.1 ANSI, 1976 B, sha `e8b2c6f4`) — superseded by SYSTAT, left in place
- `SYSTAT.BAT` (v1.1 plain paused, 925 B, sha `e85e7c…`)
UDM `192.168.0.254`: restarted `unifi.service` (Network app; PID 25420 → 730335). Mike uninstalled UniFi Protect 7.1.83, Access 4.2.29, Talk 5.2.7 (+ freeswitch) via UI.
`errorlog.md`: 2 entries (CRLF friction; CTONW boot-hang). No repo commits pushed this session (submodule work still uncommitted).
## Credentials & Secrets
No new credentials created or discovered. Used (read-only) from vault:
- `infrastructure/gururmm-server.sops.yaml` — RMM API admin login
- `clients/dataforth/udm.sops.yaml` — UDM `192.168.0.254`; SSH root key auth from GURU-5070 works (BatchMode succeeded); admin `azcomputerguru` (2FA push on web); MongoDB `127.0.0.1:27117/ace`
- AD2 rsync-daemon password read at runtime from AD2's own `Sync-FromNAS-rsync.ps1` (`$RSYNC_PASSWORD`), never returned off-box; NAS rsync `rsync://rsync@192.168.0.9/test`
## Infrastructure & Servers
- **AD2** `192.168.0.6` — Dataforth DC, GuruRMM agent **re-enrolled: new id `4b9fe588-2a86-4855-912e-d90cf5f5e3e4`** (old `cfa93bb6-…` now stale), site "D2". cwRsync at `C:\ProgramData\chocolatey\bin\rsync.exe`.
- **D2TESTNAS** `192.168.0.9` — DF Linux storage/NAS + UDM jump host. RMM agent hostname `ix.azcomputerguru.com`, id `4ad2e426-b03f-4c5d-817c-c8c675ba73a0`, site "DF Server Storage", os linux. rsync daemon module `test` → `/data/test`.
- **UDM** `192.168.0.254` — Dataforth gateway/firewall, UbiOS, 4 GB RAM. Was running Network + Protect + Access + Talk; `unifi.service` = Network app (Java, `-Xmx512M`); `mongo ace` on `127.0.0.1:27117`. FreePBX SIP forward in `/data/on_boot.d/30-freepbx-sip-forward.sh` (→ 192.168.100.2). Reached from GURU-5070 via Tailscale (D2TESTNAS subnet router) — that route dropped when the UDM management went down.
- **GuruRMM** API `http://172.16.3.30:3001`. 14 active DOS stations: TS-1L/1R, TS-3L/3R, TS-4L/4R, TS-8L/8R, TS-10L/10R, TS-11L/11R, TS-12, TS-27 (no RMM agents — DOS).
## Commands & Outputs
- Deploy pattern: `curl POST $RMM/api/agents/<id>/command` (jq-built payload) → poll `/api/commands/<id>`; PowerShell decodes gzip+base64 batch content, sha-verifies, `rsync -v -t` push to `rsync://rsync@192.168.0.9/test/COMMON/ProdSW/`.
- Line-ending truth check: `tr -cd '\r' < f | wc -c` must equal `tr -cd '\n' < f | wc -c`; `head -c 60 f | xxd` should show `0d 0a`. NOT `grep -c $'\r$'` (false-positived).
- SPECVER verification: all 14 `.SPV` files show `5BMAIN DAT 83,200 06-26-26 11:43a`.
- UDM diagnosis: `server.log` full of `transformer has been closed` / `WebSocket session has been closed`; `free -m` ~1 GB swap used; `systemctl restart unifi` → clients repopulated, `mongo ace db.user.count()`=300.
- UDM apps: `dpkg -l` → unifi-protect 7.1.83, unifi-access 4.2.29, unifi-talk 5.2.7, freeswitch-unifi-talk; footprint Talk 1.1 G / Protect 242 M / Access 19 M; Protect 0 recorded video files.
- UDM post-uninstall: from D2TESTNAS RMM shell, `ping 192.168.0.254` = DOWN, SSH timed out — but DF machines checking into RMM (data plane up).
## Pending / Incomplete Tasks
- **UDM:** re-probe via D2TESTNAS RMM until SSH/ICMP returns; then confirm Protect/Access/Talk gone and memory recovered, and Network app came up clean. (Was mid post-uninstall reboot: forwarding up, management down.)
- **SYSTAT/CTONW v5.3:** re-dispatch the promote to the NEW AD2 id `4b9fe588…` (old dispatch stale); confirm SYSTAT paused page appears before menux.
- **Commit submodule work:** the 8 batch drafts + `.gitattributes` + `docs/` in `projects/dataforth-dos/` are uncommitted (plain sync skips submodule internals). Commit inside the submodule, then `--with-submodules` sync to bump the pointer.
- **#32489 ticket:** John reply drafted (customer-visible, held per Mike until all changes done); decide status (Resolved vs keep open) and **billing** (Mike chose to bill — needs minutes + labor type; check Dataforth prepaid block).
- **CONFIG.SYS/ANSI:** additive `DEVICE=C:\DOS\ANSI.SYS` per station (preserving hardware lines) if pursuing ANSI, not a master file.
- **UDM memory (root cause):** removing Protect/Access/Talk is the durable fix; verify RAM headroom post-removal so the 0-clients doesn't recur.
- Clean stray extension-less files in `T:\STATUS` (from the v5.1 incident).
## Reference Information
- Syncro **#32489** (id 113201089), Dataforth Corp (customer 578095), contact John Lehman (2851723), was "Customer Reply", past due 2026-07-02.
- Prior audit: `clients/dataforth/docs/audits/2026-07-01-test-data-chain-audit-AD2.md`; project: `projects/dataforth-dos/`.
- Batch source of truth: `projects/dataforth-dos/batch-files/`. Tier-A design: `projects/dataforth-dos/docs/TIER-A-STATION-TELEMETRY.md`.
- Boot flow: `AUTOEXEC → STARTNET → NWTOC (download) → CTONW (upload) → menux`. NWTOC pulls `COMMON\ProdSW\*.BAT` → `C:\BAT` each boot (self-update path; broken v5.0 `/Y` cannot bootstrap v5.1).
- SHAs above are file content hashes (sha256) used for deploy verification, not git commits.