claudetools/clients/dataforth/session-logs/2026-07/2026-07-01-mike-dataforth-test-data-chain-audit.md

# Dataforth test-data-chain audit via RMM-spawned AD2 Claude + multi-AI verification

## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin

## Session Summary

Ran a full audit of the Dataforth test-data chain in support of Syncro #32489 (DOS test
stations not pulling updated spec files). The pivotal enabler was proving a new capability:
**spawning a headless `claude -p` on the AD2 domain controller via its GuruRMM agent** rather
than the async git-sync handoff. AD2 (192.168.0.6) is isolated from the ACG coord API but its
RMM agent phones home, and it is online (agent `cfa93bb6-...`). A staged probe confirmed:
sysadmin is logged into the console (so `context: user_session` works and returns an elevated
token), Claude Code v2.1.181 is at `C:\Users\sysadmin\.local\bin\claude.exe`, node v20 is
system-wide, and the repo is at `C:\ClaudeTools`. Headless `claude -p` initially failed with
`Invalid API key` because a stale machine-level `ANTHROPIC_API_KEY` (108 chars) shadowed
sysadmin's good OAuth creds; unsetting it (`Remove-Item Env:\ANTHROPIC_API_KEY`) let it fall
back to `.credentials.json` and return `PROBE_OK`.

With the pattern proven, launched a strictly read-only autonomous Claude auditor on AD2,
detached (runner pid 4748) so it survived the RMM command-timeout window, writing to
`C:\Users\sysadmin\ad2-audit\FINDINGS.md` with a `DONE.txt` marker. A background monitor polled
the marker; the run completed in ~17 min (exit 0, 25 KB report). The report is excellent,
evidence-backed, and corrected several stale assumptions in `CLAUDE.dataforth.md` (dated
2026-03-29).

Key findings: **F1 (HIGH)** — deployed `NWTOC.BAT` v5.0 (`COMMON\ProdSW`, verified on AD2 and
NAS) copies only `*.BAT` and `*.EXE`, **zero `.DAT`** — the confirmed root cause of #32489; and
no NWTOC version ever distributed the shared `COMMON` masters, so v5.1 must ADD a DATA copy, not
restore one. **F2 (HIGH, new)** — `NWTOC.BAT` v5.0 and `CTONWTXT.BAT` v2.3 use `COPY /Y`, which
is not a valid MS-DOS 6.22 switch (introduced in MS-DOS 7.0/Win95); on true 6.22 this errors and
copies nothing, meaning NWTOC may be failing entirely. Stale-assumption corrections: datastore is
now **PostgreSQL 18** (SQLite is a 4.4 GB archive), the scheduled task runs
**`Sync-FromNAS-rsync.ps1`**, web delivery is a live HTTP API uploader (472,290 records flagged,
not the dead `For_Web`), and `CTONWTXT` IS invoked. Also F3 (stray `TS-21\ProdSW` file breaks
rsync push every run), F4 (server generates from a frozen 2026-03-27 specdata snapshot), F5
(plaintext creds in scripts).

Then ran the requested multi-AI cross-verification of the load-bearing DOS-6.22 claim. **Grok**
(verify mode) independently confirmed F2 in full: 6.22 `COPY` supports only `/A /B /V`; `/Y`
arrived in MS-DOS 7.0/Win95; `COPY /Y src dst` → `Invalid switch - /Y`, 0 files copied; plain
`COPY` overwrites silently on 6.22 (the correct form). **Gemini** was unavailable this session
(quota exhausted on `gemini-3.1-pro-preview` + an OAuth fallback error; logged). The pivotal
unresolved question is empirical and station-only: do the stations run genuine 6.22 (→ NWTOC has
been copying nothing since 2026-03-16) or MS-DOS 7.x (→ `/Y` is fine, F1 is the sole cause)?
Stations have no RMM agent (they are DOS), so `VER` + a `COPY /Y` test must be done on a station.

## Key Decisions

- Used RMM-spawned headless Claude on AD2 (not the sync handoff) for live ground truth, since
  the committed docs were 3 months stale and AD2's RMM agent is reachable despite coord isolation.
- Ran the AD2 agent strictly READ-ONLY with an ironclad brief (no writes/git/state changes,
  deliverable file only) and detached + polled, given it runs on a production domain controller.
- Cross-verified only the highest-value, easy-to-get-wrong claim (DOS-6.22 `COPY /Y`) with a
  second vendor, rather than re-running a redundant Claude fan-out over an already-thorough report.
- Did NOT apply any fix or touch #32489 yet — the v5.1 design and severity both hinge on the
  unresolved station DOS-version question.

## Problems Encountered

- Headless `claude -p` on AD2 returned `Invalid API key` — a stale machine `ANTHROPIC_API_KEY`
  shadowed the OAuth creds. Fixed by unsetting the env var before invoking (OAuth fallback).
- RMM dispatch failed once with `/mingw64/bin/curl: Permission denied` (transient AV lock on the
  Git-Bash curl); nothing dispatched. Retried using `/c/Windows/System32/curl.exe`.
- Gemini (`agy`) verify failed — `gemini-3.1-pro-preview` quota exhausted and the default-model
  fallback errored on OAuth `_doSetupUser`. Logged; used Grok as the cross-vendor check.

## Configuration Changes

Created:
- `clients/dataforth/docs/audits/2026-07-01-test-data-chain-audit-AD2.md` — the full AD2 audit
  report (pulled back via RMM, gzip+base64).
- `.claude/memory/reference_rmm_spawn_headless_claude.md` — the validated RMM-spawn-Claude
  pattern + the ANTHROPIC_API_KEY-shadow gotcha.
- MEMORY.md index line for the above.
- On AD2 (NOT in repo): `C:\Users\sysadmin\ad2-audit\{brief.txt,run.ps1,run.log,FINDINGS.md,DONE.txt}`.

Modified:
- `errorlog.md` — logged the Gemini unavailability.

## Credentials & Secrets

- No new credentials created. The AD2 audit surfaced pre-existing plaintext creds in Dataforth
  scripts (F5) — flagged for rotation + vaulting, NOT copied anywhere: rsync daemon
  (`rsync`/`IQ2...19`) in `Sync-FromNAS-rsync.ps1`; NAS root SSH password in the dormant
  `Sync-FromNAS.ps1`; Postgres `testdatadb_app` password in `testdatadb\database\db.js`.
- AD2 has a stale machine-level `ANTHROPIC_API_KEY` (invalid) that must be unset before running
  headless Claude there; sysadmin's OAuth creds at `C:\Users\sysadmin\.claude\.credentials.json`
  are the working auth.

## Infrastructure & Servers

- AD2: `192.168.0.6`, Dataforth DC, Windows Server 2019 (10.0.17763), RMM agent
  `cfa93bb6-0cdc-4d4e-a29e-1609cda6f047`, client "Dataforth Corp". claude.exe v2.1.181,
  node v20.10.0, repo `C:\ClaudeTools` (branch ad2).
- NAS D2TESTNAS `192.168.0.9` (Linux/Samba), rsync daemon port 873 module `test` → `/data/test`.
- TestDataDB: PostgreSQL 18.3 service `postgresql-18` on AD2 (`::1:5432`), db `testdatadb`
  (test_records 475,553; work_orders 34,149). App at `C:\Shares\testdatadb\`.
- Shares on AD2: `C:\Shares\test\` (COMMON\ProdSW deployed batch set; Ate\ProdSW\<type>DATA master
  specs), `C:\Shares\webshare\` (Test_Datasheets active; For_Web dead since 2026-05-11).
- Engineering master `5BMAIN.DAT` 83,200 B mtime 2026-06-26 present on AD2 + NAS; server-side
  `testdatadb\specdata\5BMAIN.DAT` is a frozen 2026-03-27 copy (F4).

## Commands & Outputs

- RMM dispatch to AD2 with `context:user_session`, `timeout_seconds` (not `timeout`), via
  `/c/Windows/System32/curl.exe`. Headless launch: detached `Start-Process powershell -File
  run.ps1 -WindowStyle Hidden`; runner unsets `ANTHROPIC_API_KEY`, `cd C:\ClaudeTools`, runs
  `claude -p <brief> --permission-mode bypassPermissions --output-format text`.
- Probe result: `PROBE_OK` after unsetting the env key (exit 0, 13s).
- Audit result: `DONE exit=0`, FINDINGS.md 25,337 B, ~17 min.
- Grok verify verdict: claim correct; nitpicks = "Invalid switch" not "Invalid parameter", and the
  error is visible (not literally "silent").

## Pending / Incomplete Tasks

- **PIVOTAL:** confirm station DOS version (`VER` + `COPY /Y NUL C:\TEST.TXT` on a station).
  Determines whether F2 is catastrophic (6.22 → NWTOC copies nothing since 2026-03-16) or moot
  (7.x). Stations have no RMM agent — reach one via NAS/console.
- Draft DOS-6.22-safe `NWTOC v5.1`: plain `COPY` (no `/Y`), one-way pull of master `.DAT`s into a
  distinct local dir (avoids the cyclic-overwrite v5.0 guarded against), 6.22-valid `IF EXIST`/
  `GOTO`. Grok-review before it touches a station. Correct on both 6.22 and 7.x.
- Update Syncro #32489 with the confirmed root cause (F1 + F2) and plan.
- Address F3 (remove stray `TS-21\ProdSW` file), F4 (feed specdata from masters), F5 (rotate/vault
  creds) — recommendations only, nothing applied.
- Retry Gemini verification later for true two-vendor triangulation.

## Reference Information

- Audit report: `clients/dataforth/docs/audits/2026-07-01-test-data-chain-audit-AD2.md`.
- Syncro #32489 (id 113201089, Scheduled, due 2026-07-02); John Lehman contact 2851723; Dataforth
  Corp customer 578095; appointment 5626864474.
- AD2 RMM agent `cfa93bb6-0cdc-4d4e-a29e-1609cda6f047`.
- Memory: `reference_rmm_spawn_headless_claude`, `gururmm-command-timeout-seconds`.