sync: auto-sync from AD2 at 2026-06-18 06:13:27

Author: Mike Swanson
Machine: AD2
Timestamp: 2026-06-18 06:13:27

.claude/memory/MEMORY.md

@@ -99,8 +99,9 @@
 - [Dashboard beta-first deploy](feedback_dashboard_beta_first.md) — Dashboard auto-builds to rmm-beta.azcomputerguru.com on push; prod (rmm.azcomputerguru.com) is explicit promote-only via promote-dashboard.sh --confirm. Never hand-rsync prod. One artifact, nginx sub_filter BETA banner. Stood up 2026-06-02.
 
 ### Cascades
-- [Cascades operational rules](feedback_cascades.md) — Active rules: (1) folder redirection (fdeploy) needs subfolders PRE-CREATED before first logon or it caches a failure forever; recovery via fix-shell-redirect.ps1. (2) ALWAYS ask which security group(s) a new user goes into — never auto-derive from OU. (3) Do NOT lock down the legacy Main\Company Web Docs\Accounting (Everyone:Full) folder — still in active use.
+- [Cascades operational rules](feedback_cascades.md) — Active rules: (1) folder redirection (fdeploy) needs subfolders PRE-CREATED before first logon or it caches a failure forever; recovery via fix-shell-redirect.ps1. (2) ALWAYS ask which security group(s) a new user goes into — never auto-derive from OU. (3) Do NOT lock down the legacy Main\Company Web Docs\Accounting (Everyone:Full) folder — still in active use. (4) NEVER change Cascades production infra (pfSense/UniFi/switches/DHCP) without discussing it + explicit per-change go — read-only/dry-run until then.
 - [Cascades FR GPO fix](reference_cascades_fr_gpo_fix.md) — Native Folder Redirection was DOA on every machine: redirect targets were in a misnamed `fdeploy1.ini` (Windows reads `fdeploy.ini`) → empty target path → silent no-op → per-user registry workaround every time. Fixed 2026-06-08 (correct fdeploy.ini + version bump). Also: CS-SERVER live RMM agent is `c39f1de7...` (old `6766e973` stale).
+- [pfSense 25.07 ops quirks](reference_pfsense_25_07_ops.md) — Cascades pfSense Plus 25.07: logs are PLAIN TEXT (use tail/grep, NOT clog → clog returns empty); clean dhcpd restart = `services_dhcpd_configure()` via slow pfSsh.php (needs 50s+ timeout); dirty boot can leave 2 dhcpd → DISCOVER/OFFER but no ACK; reboot the Cox modem after a config restore; ZFS survives power loss. From the 2026-06-17 power-outage incident.
 - [feedback_ascii_only_api_payloads](feedback_ascii_only_api_payloads.md) -- On Windows/Git-bash, non-ASCII chars (em-dash, arrow, smart quotes) in JSON payload TEXT passed to curl get mangled and rejected — Discord bot-alert returns 400, the coord API returns "error parsing the body". Use ASCII-only in API payload text, or a single-quoted heredoc.
 
 ## Machine

.claude/memory/feedback_cascades.md

@@ -1,6 +1,6 @@
 ---
 name: Cascades-specific operational rules (folder redirect, security groups)
-description: Active rules for Cascades work — (1) folder redirection (fdeploy) needs subfolders pre-created before first logon or it caches a failure forever; recovery via fix-shell-redirect.ps1; (2) always ASK which security group(s) a new user goes into — never auto-derive from OU; (3) do NOT lock down the legacy Main\Company Web Docs\Accounting (Everyone:Full) folder — still in active use. Root-cause / incident detail in project_cascades_history.md.
+description: Active rules for Cascades work — (1) folder redirection (fdeploy) needs subfolders pre-created before first logon or it caches a failure forever; recovery via fix-shell-redirect.ps1; (2) always ASK which security group(s) a new user goes into — never auto-derive from OU; (3) do NOT lock down the legacy Main\Company Web Docs\Accounting (Everyone:Full) folder — still in active use; (4) NEVER apply a change to Cascades production infra (pfSense, UniFi controller, switches, DHCP) without discussing it and getting an explicit per-change go — investigate read-only / dry-run only until then. Root-cause / incident detail in project_cascades_history.md.
 type: feedback
 ---
 
@@ -45,3 +45,13 @@ OU placement is mechanical (controls Entra Connect sync scope); group membership
 ## 3. Do NOT lock down the legacy `Main\Company Web Docs\Accounting` folder
 
 The accounting folder under the Synology-Drive-synced tree (`D:\Shares\Main\Company Web Docs\Accounting`, `Everyone:FullControl`) stays as-is — Howard confirmed 2026-06-10 the team is **still actively using it**. Do not scope/tighten its ACL or "clean it up" as a HIPAA hardening step, even though the wide-open Everyone:Full looks like an obvious target. The 2026-06-09 scan-to-folder build deliberately created a *separate* clean share (`\\CS-SERVER\AcctDept` → `D:\Shares\Accounting`) rather than reusing this folder; that is the lockdown story, and the legacy folder is intentionally left untouched.
+
+---
+
+## 4. NEVER change Cascades production infra without discussing it first
+
+Do not apply ANY change to the Cascades production network — pfSense (firewall rules, DHCP, `ping-check`, service restarts, reboots), the UniFi controller (radio power/channel/min-RSSI/disable, WLAN/PPSK settings), switches, or DHCP scopes — until it has been **discussed and explicitly approved, per change**. Investigate **read-only / dry-run only** (e.g. `apply-radio` without `--apply`, `pfsense-ssh.sh audit`/read-only `run`) and present proposals; wait for an explicit go before any write.
+
+**Why:** Howard set this explicitly 2026-06-17 during the Poly-phone-drop investigation — it's a live HIPAA assisted-living network (~780 clients, residents' medical/IoT devices) where a bad change has real patient-care and compliance impact, and changes need coordination (another session was concurrently doing radio work; Mike should be looped in on pfSense changes).
+
+**How to apply:** dry-run/read-only by default; stage changes as reviewable proposals; one explicit approval per change, not a blanket one. Pair with the per-change confirmation already required for hard-to-reverse/outward-facing actions. Coordinate via the coord API when another session may touch the same gear. Note the per-room /28 segmentation is intentional HIPAA L2 isolation — do not "clean it up." See [[project_cascades]].

.claude/memory/feedback_prefer_ssh_over_rmm.md

@@ -0,0 +1,14 @@
+---
+name: prefer-ssh-over-rmm
+description: When a machine has SSH available and SSH is easier, prioritize SSH over driving it through the GuruRMM agent
+metadata:
+  type: feedback
+---
+
+When a target machine has **SSH available** (e.g. key auth already set up) and the task is easier over SSH, **prioritize SSH over the GuruRMM agent**. Drive it with the system OpenSSH (`C:\Windows\System32\OpenSSH\ssh.exe` / `scp.exe`) — `scp` the script over, run it via `ssh`.
+
+**Why:** SSH runs as a real interactive user, which avoids RMM's pitfalls: RMM commands run as **SYSTEM** (git "dubious ownership", credential-context gaps) and are bound by the **server-side command-timeout reaper** (long jobs get killed mid-run — and a killed RMM command can leave an orphaned child process). RMM also forces base64/PowerShell-quoting gymnastics for every script. SSH sidesteps all of it.
+
+**How to apply:** for a machine with SSH, default to `scp script + ssh run` (keep SSH commands simple — cmd-native or `node absolute-path.js`; complex nested PowerShell-over-SSH quoting hangs). Reserve **RMM as the fallback** when SSH/VPN to the target is down — RMM is agent-out to the GuruRMM cloud, so it stays reachable even when a direct VPN flaps (and is the only option on machines without SSH). Confirm SSH reachability first; fall back to RMM on connect timeout.
+
+Example: Dataforth AD2 — `ssh sysadmin@192.168.0.6` (key auth, vault `[[reference_unifi_site_manager_api]]`-style entry at `clients/dataforth/ad2`). [[feedback_windows_quote_stripping]]

.claude/memory/reference_pfsense_25_07_ops.md

@@ -0,0 +1,28 @@
+---
+name: reference_pfsense_25_07_ops
+description: pfSense Plus 25.07 operational quirks learned during the Cascades power-outage recovery — plain-text logs (NOT clog), clean dhcpd restart via pfSsh.php, reboot the upstream modem after a config restore, ZFS power-loss resilience
+metadata:
+  type: reference
+---
+
+Learned on the Cascades pfSense (`192.168.0.1`, Plus 25.07-RELEASE, ZFS) during the 2026-06-17
+power-outage recovery. Access: `bash .claude/skills/unifi-wifi/scripts/pfsense-ssh.sh cascades-tucson run "<cmd>"`
+(admin SSH = real shell). Incident report: `clients/cascades-tucson/reports/2026-06-17-power-outage-incident.md`.
+
+- **Logs are PLAIN TEXT (ASCII), not clog binary.** `clog /var/log/dhcpd.log` returns EMPTY on 25.07
+  → do NOT conclude "logs are empty / service not logging." Read with `tail`/`grep`/`cat` directly.
+  (Burned a whole hypothesis on this — the DHCP server was actually fine.) `file /var/log/*.log` → ASCII text.
+- **Clean single-instance DHCP restart from shell:** `echo "services_dhcpd_configure();" | /usr/local/sbin/pfSsh.php`
+  (regenerates `/var/dhcpd/etc/dhcpd.conf` + restarts ONE dhcpd; kills duplicates). A power-loss/dirty boot
+  can leave **two `dhcpd` processes** fighting → clients get DISCOVER→OFFER but never REQUEST/ACK.
+  Verify: `pgrep -f "dhcpd -user" | wc -l` should be **1**. Test config: `dhcpd -t -cf /var/dhcpd/etc/dhcpd.conf`.
+- **`pfSsh.php` is SLOW to load (~20-40s).** SSH commands that invoke it need a long timeout (50s+) or they
+  time out mid-run and you can't tell if the action took.
+- **After a pfSense config restore/replace, REBOOT the upstream modem** (Cox at Cascades) to re-sync the WAN —
+  skipping this prolongs post-restore issues. Add to any restore runbook.
+- **ZFS root is power-loss resistant** — `zpool status -x` → "all pools are healthy"; `config.xml` survived an
+  unclean power-off intact. A 50x on the GUI right after a dirty boot is usually transient (services still starting).
+- **DHCP "offers but never completes" on ONE segment/switch** = asymmetric L2 forwarding (DISCOVER reaches
+  pfSense + OFFER sent on the right iface/subnet, but REQUEST=0/ACK=0 because the reply doesn't reach the client).
+  Root cause is the switch (re-adopted with stale forwarding/bad port profile), NOT pfSense — fix = reset/re-adopt
+  that switch. See [[reference_cascades_fr_gpo_fix]] for other Cascades infra notes.

clients/cascades-tucson/reports/2026-06-17-power-outage-incident.md

@@ -0,0 +1,105 @@
+# Incident Report — Cascades of Tucson Site-Wide Network Outage (Power Outage)
+
+- **Date:** 2026-06-17
+- **Client:** Cascades of Tucson
+- **Severity:** High — total network outage (all APs, switches, clients) at a 24/7 HIPAA assisted-living facility
+- **Status:** RESOLVED — full network back online
+- **Logged by:** Howard Enos (Howard-Home). Mike Swanson worked the on-box/console recovery.
+- **Purpose:** Basis for a power-outage ticket + corrective/preventive actions.
+
+## Summary
+
+A building power outage took the entire Cascades network down. The pfSense firewall lost power
+**uncleanly** because it was plugged into the **surge-only side of the UPS, not the battery-backed
+side**, so it hard-powered-off instead of riding through / shutting down gracefully. The unclean
+shutdown led to a messy recovery: a **duplicate `dhcpd` process** on pfSense and **broken Layer-2
+forwarding on a 2nd-floor switch** that prevented APs/clients from completing DHCP. Recovery was
+further delayed because **the Cox modem was not rebooted after the pfSense config restore**.
+Restoring service required: re-seating pfSense onto the battery side (done), a clean pfSense config
+restore from on-box auto-backups, eliminating the duplicate dhcpd, **resetting + re-adopting the
+2nd-floor #2 switch** (which brought floors 2/3/4 back), and the Cox modem reboot.
+
+## Impact
+
+- **All 77 APs + all 12 switches showed disconnected; 0 clients; no DHCP** for the duration.
+- Resident/staff WiFi, the Poly VoIP phones, and wired devices were down site-wide.
+- Duration: power-loss through full restoration (afternoon/evening 2026-06-17).
+
+## Root Causes
+
+1. **Primary trigger — building power outage.**
+2. **pfSense had NO battery protection** — the firewall was connected to the **surge-only outlets
+   of the battery backup, not the battery-backed outlets**, so it lost power instantly and shut
+   down uncleanly (the rest of the protected gear behaved differently). **RECTIFIED** — pfSense
+   moved to the battery-backed side.
+3. **Unclean shutdown → degraded recovery on pfSense:** a **duplicate `dhcpd` instance** came up.
+   Two dhcpd processes fought over the same scopes, so clients got `DISCOVER → OFFER` but the
+   handshake never completed (`REQUEST/ACK` failed) → no IPs.
+4. **2nd-floor switch L2 forwarding broke on re-adoption:** the switch came back "online" to the
+   controller (management/uplink OK) but **forwarded AP DHCP requests UP to pfSense while not
+   delivering pfSense's replies back DOWN to the APs** — an asymmetric/one-way forwarding failure
+   (stale forwarding state / port-profile not cleanly applied after the dirty power-up).
+5. **Cox modem not rebooted after the pfSense restore** — after restoring the pfSense config, the
+   upstream Cox modem was not power-cycled, so the WAN did not fully re-establish, prolonging the
+   issues post-restore. (Modems commonly bind to the firewall's WAN MAC/DHCP state and need a
+   reboot after a firewall restore/replace.)
+
+## Detection & Diagnosis (key findings)
+
+- Controller (independent of the remote VPN) showed **77/77 APs + 12/12 switches disconnected,
+  0 clients** — confirming a real, common-path outage, not a monitoring artifact.
+- pfSense answered ICMP (`192.168.0.1` up) but its **SSH/management hung** initially and the GUI
+  briefly returned a **50x** (transient, services still starting after the unclean power-up).
+- **ZFS pool healthy** (`all pools are healthy`) and **`config.xml` intact** (684,842 bytes, valid)
+  — no filesystem/config corruption; ZFS's copy-on-write protected it.
+- Reading logs **directly** (pfSense 25.07 uses plain-text logs, not `clog`): DHCP server itself was
+  healthy (leases writing; clients that reached REQUEST got ACKed) — but APs/native-LAN clients were
+  stuck at OFFER. DORA breakdown for the 16 second-floor APs: **404 DISCOVER → 404 OFFER → 0 REQUEST
+  → 0 ACK**, offers correctly on `igc1` / `192.168.2-3.x` — proving pfSense was offering correctly
+  and the **replies were not reaching the APs** (downstream switch forwarding).
+- Found **2 `dhcpd` processes** running (should be 1) — the duplicate explaining the broken completion.
+
+## Resolution (what fixed it)
+
+1. **pfSense re-seated onto the battery-backed UPS outlets** (corrective; prevents recurrence of #2).
+2. **pfSense config restored** from the on-box auto-backup history (`/cf/conf/backup/config-*.xml`);
+   ZFS/config were intact so this was clean.
+3. **Duplicate `dhcpd` eliminated** → single clean instance via `services_dhcpd_configure()`; DHCP
+   began completing (switches + Poly phones started getting IPs).
+4. **Reset + re-adopted the 2nd-floor #2 switch** (`Switch 2nd Floor #2`, USL24PB, 192.168.2.193).
+   This cleared the broken forwarding state; **floors 3 and 4 came up immediately afterward**
+   (they were downstream of it), and the site returned to fully online.
+5. **Cox modem rebooted** after the restore to re-establish the WAN.
+
+## Corrective Actions
+
+- [DONE] pfSense moved from the **surge-only** side to the **battery-backed** side of the UPS.
+
+## Preventive Actions / Follow-ups (for the ticket)
+
+1. **Enable Netgate AutoConfigBackup on pfSense** — there was **NO off-box config backup** at
+   incident time, which turned a blip into a scramble. ACB gives automatic encrypted cloud backups
+   on every change. (An off-box copy of the current + pre-outage config was captured locally during
+   the incident; should also be vaulted.)
+2. **Verify full UPS coverage + runtime + clean shutdown:** the **NUT** package is configured on
+   pfSense, but the box still went down hard (it was on surge-only). Confirm that pfSense AND the
+   **core/aggregation switch + the AP-PoE switches** are all on **battery-backed** outlets with
+   adequate runtime, and test that NUT triggers a graceful pfSense shutdown on power loss.
+3. **Post-restore runbook step:** after any pfSense config restore/replacement, **reboot the Cox
+   modem** to re-sync the WAN (this was the missed step that prolonged recovery).
+4. **Vault the pfSense config** off-box (immediate safety net beyond ACB).
+5. **Optional monitoring:** alert on DHCP-not-completing / duplicate-dhcpd / mass device-disconnect
+   so a future event is caught in minutes.
+
+## Reference Information
+
+- **pfSense:** `192.168.0.1`, Plus 25.07-RELEASE, ZFS. WAN: Cox (`igc0`, 184.191.143.x). LAN: `igc1`
+  192.168.0.0/22 + per-room /28 VLANs (HIPAA isolation) + VLAN20 (`10.0.20.0/24`) + VLAN30 voice.
+  Admin cred vault `clients/cascades-tucson/pfsense-firewall`. Access via
+  `unifi-wifi/scripts/pfsense-ssh.sh cascades-tucson`.
+- **Switch that needed reset+re-adopt:** `Switch 2nd Floor #2` (USL24PB) `192.168.2.193`. Also
+  present: `Switch 2nd Floor` (US24PRO) `192.168.2.4`.
+- **UniFi controller:** UOS `172.16.3.29`, Cascades site `va6iba3v` / `685f39068e65331c46ef6dd2`.
+- **Fleet:** 77 APs, 12 switches, ~573–788 clients. 22 Poly WiFi phones (OUI `48:25:67`, VLAN 20).
+- **Key technical notes:** pfSense 25.07 logs are **plain text** (use `tail`/`grep`, NOT `clog`).
+  Clean DHCP restart on this box = `services_dhcpd_configure()` via `pfSsh.php`.

clients/cascades-tucson/session-logs/2026-06/2026-06-17-howard-cascades-poly-phone-drops-network-smoothing.md

@@ -0,0 +1,93 @@
+# Cascades — Poly phone-drop root cause + whole-network smoothing plan (IN PROGRESS, no changes applied)
+
+- **Date:** 2026-06-17
+- **Machine:** Howard-Home
+- **Client:** Cascades of Tucson
+- **Status:** Investigation complete; remediation PLANNED + dry-run verified; **NOTHING APPLIED**. Resume at "Pending / Next steps".
+
+## User
+- **User:** Howard Enos (howard)
+- **Machine:** Howard-Home
+- **Role:** tech
+
+## Session Summary
+
+Started as a CS-SERVER failing-drive review (separate log: `2026-06-17-howard-cs-server-drive-review-and-spike-question.md`), pivoted to a Cascades WiFi "noon network spike" question, then to a live complaint that the **Poly WiFi phones are dropping** ("around 102"). Ran a deep, multi-angle investigation against the UniFi controller (UOS `172.16.3.29`, Cascades site `va6iba3v` / `685f39068e65331c46ef6dd2`) via `.claude/scripts/uos-mongo.sh`, and pfSense (`192.168.0.1`, Plus 25.07) via `unifi-wifi/scripts/pfsense-ssh.sh cascades-tucson` (admin SSH; VPN already up — pfSense reachable). All read-only / dry-run.
+
+**Root cause of the phone drops = the INTENTIONAL pfSense reboot on 6/16 at 22:38:12 MST.** Built a per-phone drop-rate timeline from `ace_stat.stat_hourly` (o:'user', `duration` field is in MILLISECONDS): the 30 Poly phones run ~99.9% connected every day EXCEPT 6/16 (uptime 97.95%, 28 drop-hours) — one fleet-wide event where 28 of 30 phones each dropped exactly once, on every floor including **Floors 5/6 which the radio work left untouched**. Only a gateway-level event (the pfSense reboot, kern.boottime = Tue Jun 16 22:38:12 2026) explains all-floors-at-once; the 2.4 radio apply cannot (it only reprovisions touched APs). Today (6/17) the phones are back to ~99.77% / 1 blip. Howard confirmed the reboot was intentional → phone mystery CLOSED. The complaint came in today but today's data is healthy; goal reframed to **smooth the entire network for all devices**.
+
+Corrected several of my own earlier wrong reads during the session (logged honestly): (1) the site-wide `sta_dhcp_failures` spike since 6/14 is DECOUPLED from the phones (phones were 99.99% on 6/14–15); (2) DHCP is HEALTHY — my "empty DHCP log" was a tooling error: pfSense 25.07 uses PLAIN-TEXT logs, I was reading with `clog` (old binary format); read directly, dhcpd.log shows 1241 ACK / 1 NAK / 0 no-free-leases; (3) band steering + min-RSSI are PRE-EXISTING, not the new trigger; (4) the per-room /28 subnets are INTENTIONAL HIPAA isolation and are healthy (fullest 12/13), not the cause.
+
+Produced a prioritized whole-network smoothing plan, dry-ran the radio changes, and answered Howard's questions on client limits and the residents bandwidth cap. Howard set a hard rule: **do not change anything on Cascades prod infra without discussing it + explicit per-change go** (saved to memory `feedback_cascades.md` rule #4 + MEMORY.md). Saving now to survive context condensation, then continuing.
+
+## Key Decisions / Findings
+
+- **Phone drop root cause = intentional 6/16 22:38 pfSense reboot** (one-time, transient; resolved). NOT the 2.4 radio work, NOT DHCP, NOT the 6/14 DHCP-metric spike.
+- **DHCP server is healthy** (read logs DIRECTLY, not via clog). `sta_dhcp_failures` metric is client/WiFi-side (frames lost at 100% retry), not pfSense.
+- **No WLAN client-device limit is set** (CSCNet/CSC ENT/Guest empty). Busiest AP = Dining Room 67 clients; median busy ~18; U7 Pro handles it. No limit needed; if a safety cap is wanted, ~127/radio. Not a drop cause.
+- **40 Mbps residents cap = PER-CLIENT** (`Resident` user group 40 down/10 up), not a shared VLAN pipe. Generous; NOT a drop cause; raising it won't help drops. NOTE: CSCNet's default usergroup is actually 'Default' = 100/100 (id `685f39078e65331c46ef7edb`); `Resident` group is `685f39078e65331c46ef7edc` with enabled-flag `undefined` — VERIFY whether the 40 cap is actually active/which clients before touching.
+- **Auto-action settings verified OFF/safe**: `network_optimization`=false, `ips`=IDS-only (non-blocking), `radio_ai.auto_enabled`=false, mesh=none (0 wireless uplinks), PMF/802.11r/v/k=off, group_rekey=0. No hidden auto-deauth.
+- **Per-room /28 segmentation = intentional HIPAA L2 isolation** via CSCNet shared-PPSK (~230–242 per-key→room-VLAN mappings, rooms 101–631). Healthy occupancy (fullest 12/13). DO NOT flatten.
+
+## Approved scope (Howard, this session)
+
+- **DO Phase 0 + Phase 1.** Dry-run first to verify (done — clean).
+- **SKIP item (e):** recover 2 down APs + 3 offline switches — requires onsite, deferred.
+- **SKIP DFS channel change** for now ("if the logs don't show issues then it's fine").
+- **KEEP band steering ON** ("I don't want band steering off just yet").
+- **Turn on EVERY kind of logging** available (pfSense) to solve quickly.
+- **Client limit:** none set; advised not needed (Howard asked for a number → ~127 if wanted).
+- **Residents 40 Mbps cap:** advised per-client + not a drop cause; verify-before-change.
+- **HARD RULE:** no changes without discussing + explicit per-change go; read-only/dry-run until then.
+
+## The verified change-set (dry-run clean; NOT applied)
+
+**Phase 1 — combined per-AP `radio_table` PUT (ONE bounce per AP — Howard's ask to avoid double-bounce):**
+- 2.4 `ng` `tx_power_mode` → **medium** — ONLY the 42 radios currently at `low` (EXCLUDE the 24 disabled [would re-enable them] + ~10 Floor-5/6 `auto` radios). `apply-radio ... ng power medium` blanket = WRONG (hits all 77).
+- 5 GHz `na` `ht` → **40** (80→40 MHz) — 76 radios (1 already 40). Clean, no disabled/wrong-direction hits.
+- 5 GHz `na` `min_rssi` → **‑82** — ONLY the 69 radios currently at `-77` enabled (EXCLUDE the 5 currently OFF: 615/608/505/517/622 [would enable], and 108 at -89 [would tighten]).
+- NO `na` channel/DFS change. NO band steering. NO client limit. NO cap change.
+- Tooling note: `apply-radio.sh` does ONE band+setting per reprovision = multiple bounces. To get ONE bounce/AP, BUILD A COMBINED radio_table PUT (ng power + na ht + na min_rssi in a single PUT per AP, correctly scoped per above). RW write cred IS vaulted: `infrastructure/uos-server-network-api-rw` (so `--apply` is possible once gone-ahead). apply-radio rollback auto-saves to `.claude/tmp/apply-rollback-*.json`.
+
+**Phase 0 — non-disruptive:**
+- pfSense `ping-check` → off (all 240 DHCP pools currently have ping-check true; adds ~1s/lease, causes client-side DHCP timeouts = the failure metric). Reversible.
+- Disable 3 AM AP firmware auto-upgrade (`mgmt.auto_upgrade=true, hour=3` — nightly AP reboots/drops).
+- Turn on full pfSense logging: DHCP (already on), DNS resolver (unbound), firewall (filter, incl. default-block + resident-VLAN rules), system, gateway; BUMP log rotation/retention. CAVEAT: DNS-query + firewall-pass logging is high-volume → set sane rotation, dial back chattiest after we catch the issue.
+
+## Problems Encountered (my own corrections — for linting)
+
+- **clog vs plain-text logs (pfSense 25.07):** read DHCP/system logs with `clog` → returned empty → I wrongly concluded "DHCP log empty / phones not doing DHCP." 25.07 logs are PLAIN TEXT; use `tail`/`grep` directly. Real data: dhcpd.log healthy (1241 ACK/1 NAK). WASTED a hypothesis. (Worth an errorlog `--friction`.)
+- **stat duration units:** `ace_stat.stat_hourly` o:'user' `duration` is MILLISECONDS, not seconds — first uptime calc was inflated 1000×. Corrected.
+- Conflated two metrics (site-wide DHCP-failures vs per-phone drops) → Howard caught it; they're decoupled (different populations, different dates).
+
+## Infrastructure & Servers
+
+- **UniFi UOS controller** `172.16.3.29`; Cascades site name `va6iba3v`, id `685f39068e65331c46ef6dd2`. Access: `infrastructure/uos-server-ssh-key` + `.claude/scripts/uos-mongo.sh` (db `ace` config, `ace_stat` history). Write cred: `infrastructure/uos-server-network-api-rw` (vaulted); read: `infrastructure/uos-server-network-api`.
+- **Cascades pfSense** `192.168.0.1`, Plus 25.07-RELEASE. Admin SSH cred `clients/cascades-tucson/pfsense-firewall`. Access via `bash .claude/skills/unifi-wifi/scripts/pfsense-ssh.sh cascades-tucson audit|dhcp|run "<cmd>"`. VPN (OpenVPN Connect) already up — pfSense reachable from Howard-Home. kern.boottime 2026-06-16 22:38:12 MST.
+- **WiFi:** 77 APs (mostly U7PRO), ~573–788 clients. CSCNet = shared-PPSK SSID (wlanconf `685f39078e65331c46ef7ee5`, networkconf `685f39078e65331c46ef8ac4`, ~242 PPSK keys), bands 2g+5g (no 6e), `no2ghz_oui=true` (band steering ON), usergroup `Default`(100/100). Other SSIDs: CSC ENT (`...7ee4`), Guest (`...7ee6`, VLAN 50 isolated), element (`...7ee3`).
+- **Poly phones:** OUI `48:25:67`, 30 seen (~22 active fleet), 26/30 on 5 GHz (na), 4 on 2.4 (ng), on VLAN 20 (`10.0.20.x`, network "Internal"). AudioCodes (8, wired) + Vertical-Remote desktop are the wired voice gear.
+- **pfSense DHCP:** 240 pools, 236 are /28 (13-host) per-room HIPAA VLANs; VLAN 20 = `10.0.20.0/24` range .50–.239 (190 hosts, 61 active), ping-check on ALL pools, lease 7200/86400. ISC dhcpd (Kea dormant).
+- **RF prior work (other session, in repo):** overnight 6/17 applied 24 of 76 2.4 radios DISABLED + 42 set to Low (~6 dBm); Floors 5/6 + mesh untouched. Over-thinned (retry 17→23.4%, satisfaction 39→30). Recommend Low→Medium. Reports: `reports/2026-06-16-unifi-full-audit.md`, `reports/2026-06-16-2.4ghz-remediation-runbook.md`. 5 GHz: 55/77 on DFS, 76/77 at 80 MHz; audit found 0 radar events (DFS low-risk). 2 APs down (108 + dup); 3 switches offline (2nd-Flr#2, 4th-Flr#2, USW Pro Max 16); ~25–34 gig ports linked at 100M.
+
+## Commands & Outputs (key)
+
+- Per-phone drop timeline: `ace_stat.stat_hourly` o:'user', oid in Poly macs, `duration` (MS) per hour → uptime%. 6/16 = 97.95% / 28 drop-hrs (the event); all other days 99.9%+.
+- pfSense boot: `sysctl -n kern.boottime` → 1781674692 = Tue Jun 16 22:38:12 2026 MST; uptime 18:13 (single reboot).
+- DHCP health (DIRECT read, NOT clog): `grep -c DHCPACK /var/log/dhcpd.log` → 1241; DHCPNAK → 1; "no free leases" → 0. `file /var/log/dhcpd.log` → ASCII text.
+- Dry-runs (all gated, no --apply): `apply-radio cascades ng power medium` (42 low correct; 24 disabled + 10 auto WRONG to include), `apply-radio cascades na width 40` (76 change clean), `apply-radio cascades na minrssi -82` (69 at -77 correct; 5 OFF + 108 WRONG).
+- Coord: posted hold-note to other radio session (msg id `5690f1b8...`).
+
+## Pending / Next steps (RESUME HERE)
+
+1. **Decide Phase 1 window** — it kicks every client off each AP as it reprovisions. Recommend per-zone (floor-by-floor) in an evening window (e.g. after 8pm), NOT all-at-once during the day. Howard to choose window (or accept brief per-AP blips now). NOT yet answered.
+2. **Get explicit go**, then:
+   - **Phase 0** (safe anytime): apply ping-check off + disable 3 AM auto-upgrade + enable full pfSense logging (with rotation).
+   - **Phase 1** (windowed): build + apply the COMBINED per-AP radio_table PUT (ng power medium [42] + na ht 40 [76] + na min_rssi -82 [69]), scoped exactly as above, per-zone, validate live with watch-ap before/after.
+3. Deferred: DFS change (revisit only if new logs show DFS issues), band steering (Howard wants kept ON for now), onsite item (e) down APs/switches, VLAN 30 voice cutover, residents 40 Mbps cap (verify active first), the 6/14 site-wide DHCP-failure metric (other clients — re-check with now-readable logs).
+4. HARD RULE: discuss + explicit per-change go before ANY apply. Read-only/dry-run only otherwise.
+
+## Reference
+
+- Memory: `feedback_cascades.md` rule #4 (no prod changes w/o discussing); `project_cascades_isolated_vlan_pattern.md` (the /28 HIPAA design + pfSense PHP API path).
+- Phone-mystery sibling log: `2026-06-17-howard-cs-server-drive-review-and-spike-question.md`.
+- Client wiki: `wiki/clients/cascades-tucson.md`.

clients/cascades-tucson/session-logs/2026-06/2026-06-17-howard-cascades-power-outage-recovery-and-5ghz.md

@@ -0,0 +1,149 @@
+# Cascades — 5GHz overlap analysis + Option-B plan, then POWER-OUTAGE incident recovery + config vaulting
+
+- **Date:** 2026-06-17
+- **Machine:** Howard-Home
+- **Client:** Cascades of Tucson
+- **Continuation of:** `2026-06-17-howard-cascades-poly-phone-drops-network-smoothing.md` (same session)
+- **Incident report (ticket basis):** `clients/cascades-tucson/reports/2026-06-17-power-outage-incident.md`
+
+## User
+- **User:** Howard Enos (howard)
+- **Machine:** Howard-Home
+- **Role:** tech
+
+## Session Summary
+
+Continued the Cascades whole-network-smoothing work from the phone-drops investigation. Built
+the AP-to-AP 5GHz overlap picture from the cached neighbor SNR matrix (`.claude/tmp/cascades-nbr.json`,
+6/16 AP collect) crossed with live channels: real co-channel overlap was MODERATE (25 genuinely
+co-channel pairs of 173 strong neighbor pairs), and width 80->40 alone only fixed 7 of 25 (28%) —
+the rest are same-channel collisions needing a channel re-plan. Howard then had UniFi auto-change
+the channels; recheck showed it made overlap WORSE (25->30 co-channel, new ch36 cluster on the
+three busiest APs salon/2nd-Floor-Atrium/CC-Bridge at 36-48 dB). A `channel-plan na` dry-run got
+co-channel to 0 but only by going non-DFS, which requires 40MHz width (at 80MHz non-DFS = 2 blocks).
+Conclusion: width + channel are coupled; the real fix is **Option B = 40MHz + non-DFS optimized
+channels + min-RSSI relax**, done as a combined per-AP radio_table PUT. Howard chose Option B and
+asked to park/save the 5GHz plan and validate the 2.4 (Low->Medium) plan separately.
+
+Mid-2.4-diagnostic, a **building POWER OUTAGE** took the entire Cascades network down (all 77 APs +
+12 switches disconnected, 0 clients). Spent the bulk of the session driving the recovery read-only
+from Howard-Home (VPN up; controller via UOS). Root cause chain: pfSense was plugged into the
+**surge-only side of the UPS, not the battery side**, so it hard-powered-off uncleanly. ZFS + config
+survived (pools healthy, config.xml valid). The dirty boot left a **duplicate dhcpd** (DISCOVER->OFFER
+but no REQUEST/ACK) and a **2nd-floor switch with one-way L2 forwarding** (APs' DISCOVERs reached
+pfSense, pfSense offered correctly on igc1/192.168.2-3.x, but OFFERs never returned -> 404 DISCOVER /
+404 OFFER / 0 REQUEST / 0 ACK across the 16 second-floor APs). Confirmed pfSense was healthy (full
+health check: services up, WAN up, no interface errors, no storm, state table fine). Killed the
+duplicate dhcpd and restarted a single clean instance via `services_dhcpd_configure()`. Mike worked
+the on-box/console side: re-seated pfSense onto the battery outlets (rectified), restored config from
+the on-box auto-backup (the 12:20 version, WITH today's VLAN30 work), **reset+re-adopted the 2nd-floor
+#2 switch** (which brought floors 3/4 up), and rebooted the Cox modem (the missed post-restore step).
+Network fully restored.
+
+Closed out by writing the incident report (ticket basis), capturing reusable pfSense-25.07 lessons
+to memory + an errorlog `--friction` entry (clog-vs-plaintext-logs), and **vaulting the pfSense
+config** (verified VLAN30 present first). While vaulting, `vault check` caught a PRE-EXISTING
+plaintext credential (`synology-signin-portal.sops.yaml`) committed plaintext in vault history
+(commit 1fbc0e1) — encrypted it go-forward; flagged the credential as exposed (needs rotation).
+
+## Key Decisions
+
+- **5GHz: go with Option B (40MHz + non-DFS optimized channels + min-RSSI relax)** as one combined
+  per-AP radio_table PUT, because width and channel are coupled — width alone is weak (7/25 pairs)
+  and a clean non-DFS channel plan requires narrowing to 40MHz. Parked pending execution (no change applied).
+- **Measured before acting on 5GHz** — the neighbor-matrix data overrode the theory (overlap moderate,
+  width change weak, auto-channel made it worse), exactly what Howard asked for.
+- **Recovery driven READ-ONLY** while Mike owned the on-box restore; the only pfSense change made was
+  killing the duplicate dhcpd + clean restart (the DHCP fix Mike was waiting on).
+- **Did NOT blind-reboot pfSense** (it was the up edge; problem was downstream). Diagnosed the 2nd-floor
+  switch as the L2 culprit; Mike's reset+re-adopt of switch #2 confirmed it (floors 3/4 followed).
+- **Vaulted the post-restore LIVE config** (not the incident-time copy), after verifying VLAN30 present.
+- **Encrypted the pre-existing plaintext synology entry** rather than leaving it — vault `check` is the
+  mandated pre-commit gate; flagged the historical exposure for rotation.
+
+## Problems Encountered
+
+- **POWER OUTAGE** -> total site outage. Root cause: pfSense on the UPS surge-only side (no battery) ->
+  unclean shutdown. RECTIFIED (moved to battery side by Mike).
+- **Duplicate dhcpd** after dirty boot -> DHCP offered but never completed. Fixed: `killall dhcpd` +
+  `echo "services_dhcpd_configure();" | /usr/local/sbin/pfSsh.php` -> single clean instance; ACKs resumed.
+- **2nd-floor switch one-way L2 forwarding** (online to controller, but not passing OFFERs down to APs).
+  Fixed by Mike: reset + re-adopt of `Switch 2nd Floor #2` (USL24PB) -> floors 3/4 recovered.
+- **Cox modem not rebooted after the pfSense restore** -> WAN didn't fully re-establish; prolonged issues.
+  Fixed by rebooting the modem. (Now a runbook step.)
+- **clog on pfSense 25.07 returned empty** -> I wrongly concluded "DHCP log empty / dhcpd not serving."
+  25.07 logs are PLAIN TEXT; read with tail/grep. Cost a hypothesis. Logged `--friction`.
+- **`pfSsh.php` is slow to load (~20-40s)** -> several restart commands timed out mid-run (timeout 60s
+  insufficient). Use 50s+ and verify after.
+- **sops encrypt failed from outside the vault dir** ("no creation rules") leaving a 730KB PLAINTEXT
+  config on disk briefly. Fixed by replicating vault-helper's pattern: `( cd "$VAULT_DIR" && sops --encrypt --in-place <relpath> )`.
+- **Pre-existing plaintext vault entry** (`synology-signin-portal.sops.yaml`) found by `vault check`;
+  encrypted it. It was committed plaintext at vault commit 1fbc0e1 -> credential in git history (exposed).
+
+## Configuration Changes
+
+- **pfSense (192.168.0.1):** killed duplicate dhcpd + clean restart via `services_dhcpd_configure()`
+  (operational, no config.xml change). All other pfSense access was read-only.
+- **pfSense hardware (Mike):** moved from UPS surge-only outlets to battery-backed outlets.
+- **pfSense config (Mike):** restored from on-box auto-backup (12:20 version, VLAN30 intact). Cox modem rebooted.
+- **UniFi (Mike):** reset + re-adopted `Switch 2nd Floor #2` (USL24PB, 192.168.2.193).
+- **UniFi channels (Howard, earlier):** auto-channel reassignment on 5GHz (made co-channel overlap worse; to be re-planned via Option B).
+- **Repo:** created `clients/cascades-tucson/reports/2026-06-17-power-outage-incident.md`; created
+  memory `.claude/memory/reference_pfsense_25_07_ops.md` + MEMORY.md index line; errorlog `--friction` entry.
+- **Vault:** created `clients/cascades-tucson/pfsense-config-backup-2026-06-17.sops.yaml` (encrypted, VLAN30 verified);
+  encrypted the pre-existing-plaintext `clients/cascades-tucson/synology-signin-portal.sops.yaml`.
+
+## Credentials & Secrets
+
+- **pfSense config backup vaulted:** `clients/cascades-tucson/pfsense-config-backup-2026-06-17` (kind note,
+  config under encrypted `content:` field; decrypts cleanly; VLAN30 present). Vault commit 3735119.
+- **[EXPOSED — ROTATE] Cascades Synology Cloud Signin Portal** (`account.synology.com`): the entry
+  `clients/cascades-tucson/synology-signin-portal.sops.yaml` was committed **plaintext** at vault commit
+  **1fbc0e1** and remains recoverable from vault git history. Now encrypted go-forward, but the credential
+  should be considered exposed and rotated. Same commit also added an "MDM service account" + "WiFi CSCNet" —
+  verify those were never plaintext in history.
+- pfSense admin SSH: vault `clients/cascades-tucson/pfsense-firewall` (unchanged).
+
+## Infrastructure & Servers
+
+- **Cascades pfSense** `192.168.0.1`, Plus 25.07-RELEASE, **ZFS** (power-loss resilient; pools healthy).
+  WAN igc0 = Cox (184.191.143.x). LAN igc1 = 192.168.0.0/22 + per-room /28 VLANs + VLAN20 (10.0.20.0/24) +
+  **VLAN30 Voice opt241/igc1.30 (10.0.30.0/24, DHCP .100-.250) — VERIFIED INTACT post-restore, status active.**
+  Logs PLAIN TEXT (not clog). Clean dhcpd restart = `services_dhcpd_configure()` via pfSsh.php.
+- **Switch that needed reset+re-adopt:** `Switch 2nd Floor #2` (USL24PB) 192.168.2.193. Also `Switch 2nd Floor`
+  (US24PRO) 192.168.2.4.
+- **UniFi UOS controller** 172.16.3.29, Cascades site `va6iba3v` / `685f39068e65331c46ef6dd2`. 77 APs, 12 switches.
+- **5GHz overlap data:** neighbor SNR matrix `.claude/tmp/cascades-nbr.json` (6/16); live channels via
+  Network API (live-stats; read cred only is `-rw`, RW cred `infrastructure/uos-server-network-api-rw` vaulted).
+- **16 second-floor APs (DHCP-stuck during incident):** names 203/204/206/209/210/217/221/222/229/236/237/240/241/247/248 + 2nd Floor Atrium; UniFi OUI 0c:ea:14.
+
+## Commands & Outputs
+
+- DORA breakdown (2nd-floor APs, the L2 proof): `404 DISCOVER -> 404 OFFER -> 0 REQUEST -> 0 ACK`, offers via igc1 on 192.168.2-3.x.
+- DHCP fix: `killall dhcpd; echo "services_dhcpd_configure();" | /usr/local/sbin/pfSsh.php`; verify `pgrep -f "dhcpd -user" | wc -l` == 1.
+- Read pfSense logs DIRECTLY (NOT clog): `tail/grep /var/log/dhcpd.log`. `file /var/log/*.log` -> ASCII text.
+- ZFS health: `zpool status -x` -> "all pools are healthy".
+- VLAN30 verify: `grep "<ipaddr>10.0.30.1" config.xml` -> opt241 "Voice" igc1.30; DHCP from 10.0.30.100 to .250; `ifconfig igc1.30` inet 10.0.30.1 status active.
+- Vault file encrypt pattern: `( cd "$VAULT_DIR" && sops --encrypt --in-place <relpath> )`.
+- 5GHz overlap: 173 strong neighbor pairs; 25 co-channel @80MHz (auto-change pushed to 30); width 80->40 fixes only 7; channel-plan na -> 0 (non-DFS, needs 40MHz).
+
+## Pending / Incomplete Tasks
+
+- **Execute Option B (5GHz)** when ready: combined per-AP radio_table PUT = ng power medium (42 low radios) +
+  na ht 40 (76) + na non-DFS optimized channels (channel-plan na) + na min_rssi -82 (69). Per-zone, evening window. Dry-run staged; nothing applied.
+- **2.4 Low->Medium bump** still pending validation/execution (the diagnostic was interrupted by the outage).
+- **Incident follow-ups (for the ticket):** enable Netgate AutoConfigBackup (no off-box backup existed);
+  verify full UPS coverage/runtime/clean-shutdown (NUT installed but box was on surge-only; confirm core +
+  PoE switches are battery-backed); add "reboot Cox modem after pfSense restore" to the runbook.
+- **[SECURITY] Rotate the Cascades Synology signin credential** (exposed in vault history, commit 1fbc0e1);
+  verify the MDM service account + CSCNet entries from that commit were never plaintext.
+- Standing rule: no Cascades prod-infra changes without discussing + explicit per-change go (memory feedback_cascades #4).
+
+## Reference Information
+
+- Incident report: `clients/cascades-tucson/reports/2026-06-17-power-outage-incident.md`
+- Memory: `reference_pfsense_25_07_ops.md` (new), `feedback_cascades.md` #4 (no-change-without-discussing).
+- Vault: `clients/cascades-tucson/pfsense-config-backup-2026-06-17` (new, commit 3735119);
+  `clients/cascades-tucson/synology-signin-portal` (encrypted; exposed in history commit 1fbc0e1).
+- pfSense access: `bash .claude/skills/unifi-wifi/scripts/pfsense-ssh.sh cascades-tucson run "<cmd>"`.
+- Prior session log (phone drops + smoothing plan): `2026-06-17-howard-cascades-poly-phone-drops-network-smoothing.md`.

errorlog.md

@@ -17,6 +17,12 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure ·
 
 <!-- Append entries below this line -->
 
+2026-06-18 | Howard-Home | pfsense-ssh/logs | [friction] used clog on pfSense 25.07 logs (now plain-text ASCII) -> empty output -> wrongly concluded DHCP log was empty / dhcpd not serving; cost a hypothesis. Read pfSense 25.07 logs with tail/grep/cat directly, NOT clog [ctx: ref=reference_pfsense_25_07_ops client=cascades-tucson]
+
+2026-06-18 | AD2 | vault | real vault.sh not found at resolved vault_path; vault read failed [ctx: path=D:/vault/scripts/vault.sh]
+
+2026-06-18 | AD2 | vault | real vault.sh not found at resolved vault_path; vault read failed [ctx: path=D:/vault/scripts/vault.sh]
+
 2026-06-17 | GURU-5070 | mailbox/365-mail | [correction] claimed in a prior session that /mailbox skill + memories were repointed off the deleted fabb3421 to the 365-mail suite, but mailbox.md still hardwired fabb3421 (token 401 AADSTS700016). Correct app is the dedicated ComputerGuru Mailbox app 1873b1b0 via get-token.sh 'mailbox' tier (cert auth); repointed mailbox.md + feedback_365_remediation_tool.md 2026-06-17. Lesson: verify the edit actually landed before reporting it done.
 
 2026-06-17 | Howard-Home | wiki-compile/coord | [friction] skill doc Phase 6 shows 'lock release claudetools wiki/<type>/<slug>' but coord.py takes 'lock release <id>'; wasted a round-trip. Capture the lock id from claim output and release by id. [ctx: ref=wiki-compile-skill]

projects/dataforth-dos/DATASHEET-FIX2-5-HANDOFF-2026-06-18.md

@@ -0,0 +1,61 @@
+# Dataforth Datasheet Fixes — Handoff to the AD2 Local Session (2026-06-18)
+
+**For:** the Claude session running locally on AD2 (`C:\Shares\testdatadb`).
+**Why you:** the remote operator's SSH/VPN to AD2 keeps flapping; you run on the box, so you have stable access. Pick up the remaining work below. **Ref Syncro ticket #32441.**
+
+You are working on the **deployed** pipeline at `C:\Shares\testdatadb` (Node + PostgreSQL 18). The repo copies under `projects/dataforth-dos/datasheet-pipeline/implementation/` are **STALE** — edit the DEPLOYED files, reconcile the repo after.
+
+---
+
+## What's already DONE (live on the website)
+- **Fix 1 (RTD label):** `datasheet-exact.js` folds RTD (sensorNum 7) into the temperature path (`Temp. (C)`, signed). Deployed; **all ~24k RTD certs re-pushed** (8B35/DSCA34/SCM5B34/SCM5B35). Audit finding resolved.
+- **Fix 4 (encoded serials):** `parsers/multiline.js` decode rule `^[A-Z]\d{3,}-` + `raw_serial_number`; `import.js` cross-model guard. Recovered 603 units, 3 quarantined (`test_records_quarantine`). Schema added: `raw_serial_number`, `test_records_quarantine`.
+- **Fix 3 (retest latest-wins):** `import.js` conflict rule = same-model AND (FAIL OR newer-by-date OR (same-date AND `raw_data IS DISTINCT` AND higher `ingest_seq`)). Parser stamps `ingest_seq = source-mtime*1e6 + line`. Schema: `ingest_seq`. Settled + validated (old-vs-new render diff vs the 16:30 dump: no corruption).
+- Save-states on AD2: `datasheet-exact.js.bak-2026-06-17-1646`, `multiline.js`/`import.js` `.bak-2026-06-17-1713` (Fix4) and `.bak-2026-06-17-1726` (Fix3).
+
+## DISCIPLINE (follow exactly — these are customer calibration certs)
+1. **Backup first:** a `pg_dump` exists at `C:\Shares\testdatadb\_backups\testdatadb-2026-06-17-1630.dump`. **Take a FRESH `pg_dump` + a VSS shadow before any schema change or re-import.** (superuser `postgres` / `Paper123!@#`; app `testdatadb_app` / `DfTestDB2026!` — both vaulted at `clients/dataforth/testdatadb-postgres`.)
+2. **Per-file save-state:** copy any file to `<file>.bak-YYYY-MM-DD-HHMM` before editing.
+3. **Atomic, asserted patches:** prepare all string-replacements in memory, assert each matches exactly once, then write; `require()` load-check after.
+4. **Validate before WRITE; validate before PUBLISH.** Never re-push to Hoffman until the per-subtype byte-validation passes.
+5. **Re-publication:** the `testdatadb` service caches the template — **restart it** (`Restart-Service testdatadb`) after changing `datasheet-exact.js` so the live service uses the new template; do re-pushes from a **fresh node process** via `uploadBySerialNumbers(serials)`. Hoffman is idempotent (returns Unchanged/Updated/Created). Re-push in batches; the DSCA fleet is ~78k certs.
+
+---
+
+## FIX 2 — DSCA Final-Test rebuild (Defect B). STAGE 1 done; do STAGE 2–3.
+
+**Root cause:** `datasheet-exact.js` has ONE hardcoded `DATA_LINES['DSCA']` + a single DSCA branch in `buildTSpecs()`. Real DSCA modules have **26 distinct Final-Test layouts** (per subtype), so names/specs/row-alignment are wrong and lines drop (e.g. Output Noise on DSCA38-05). The ACCURACY block also uses 5B/8B titles (`Vout (V)` + `====`) instead of DSCA's (`Output (V|mA)` + `----`).
+
+**STAGE 1 (DONE):** `C:\Shares\testdatadb\dsca-templates.json` already exists — **126 DSCA models**, each `{ "accOut": "Output (V)"|"Output (mA)", "rows": [ {"name": "...", "spec": "..."}, ... ] }`, extracted byte-accurately from the staged originals (the extractor used the `===` separator under the Final-Test header for exact column spans). Verified DSCA38-05 now has `Output Noise | <= 2000 uVrms`. **This JSON is the authoritative template source — use it, don't reverse DSCFIN.DAT.**
+
+**STAGE 2 (do this):** wire it into `templates/datasheet-exact.js`.
+1. Read the current DSCA render path FIRST: `parseRawData()` (how the Final-Test STATUS groups are parsed from `raw_data` — they're 5-per-line groups like `"PASS 28.42","PASS","PASS 252.2",...`), `buildTSpecs()` DSCA branch, the Final-Test render loop in `generateExactDatasheet()`, and the accuracy header line (~line 567, currently `inputHeader + '        Vout (V)  Vout (V)*  Error (%)  Status'` with `==========` separators).
+2. Load `dsca-templates.json` at module top. For `family === 'DSCA'`:
+   - Replace the param **names** and **specs** source: instead of `DATA_LINES['DSCA']` + `buildTSpecs` DSCA specs, use `DSCA_TEMPLATES[record.model_number].rows` (each row gives the name + spec text directly — no spec-file lookup needed for DSCA).
+   - Map the parsed `raw_data` STATUS groups **positionally** onto the template rows for the **measured value + PASS/FAIL status**. The staged template row order == the raw_data group order (same DOS source). **Reconcile the existing skip rule** (`if (status.length <= 4) continue`): rows like `240VAC Withstand`/`Hi-Pot` have NO measured value and an empty spec — they must still render (blank measured + blank spec + PASS), so don't drop them; align by position, show the value when the group carries one.
+   - Fix the ACCURACY block for DSCA: use `DSCA_TEMPLATES[model].accOut` (`Output (V)` or `Output (mA)`) in place of `Vout (V)`, and `----------` dash separators in place of `==========`. (The input column is already correct post-Fix-1.)
+   - If a model is missing from `dsca-templates.json` (no staged original): **do not guess** — skip/flag it.
+3. Save-state + atomic patch + load-check + restart the service.
+
+**STAGE 3 — validate per subtype (the gate):** render ALL DSCA certs and **byte-compare vs the staged originals, grouped by the 26 layouts**; require zero content-delta per layout before any re-push. Use a content-normalized compare (strip the leading `===` letterhead-separator line and trailing whitespace — those are the known, deferred cosmetic gaps; focus on the Final-Test + ACCURACY content matching). Report match/mismatch per layout; investigate any mismatch before publishing. Then re-push DSCA in batches via `uploadBySerialNumbers`, diff-gated, watching Updated/Unchanged.
+
+**Scope:** ~78k DSCA certs across DSCLOG. **Do NOT re-push until STAGE 3 is clean per subtype.**
+
+---
+
+## FIX 5 — Backfill 379 cryptolocker-era units (operational)
+379 units (Oct 2025–Jan 2026, 3 stations) have no surviving `.DAT` but their staged `.TXT` still exist — and those were rendered by the ORIGINAL (correct) software, so **publish them directly, don't round-trip through raw_data**.
+1. Add column `legacy_cert_text TEXT` to `test_records` (backup first).
+2. Identify the 379 (staged `.TXT` present, serial absent from DB — the `MISSING-UNITS-REPORT-FOR-JOHN-2026-06-17.md` on the `ad2` branch has the method/list; Cause 2 set).
+3. Insert them: `raw_data = NULL`, `legacy_cert_text` = the staged `.TXT` content, plus model/serial/test_date/station/overall_result parsed from the `.TXT`.
+4. Modify `render-datasheet.renderContent()` to return `legacy_cert_text` when `raw_data IS NULL`.
+5. **Validate:** cross-reference the 379 serials against the ERP/work-order system to confirm valid shipped units before publishing; spot-check rendered vs staged. Then publish.
+
+---
+
+## Minor cleanup
+- **1 Hoffman push error** during the Fix-3 publish — find which serial (check `notify` logs / re-run that batch with per-record fallback) and resolve.
+- **~420 "skipped" units** (no spec entry / model not registered in Hoffman — `upload-to-api.js` `UNREGISTERED_MODELS` + null-render skips). They're safely in the DB; decide per model whether to register in Hoffman or add spec coverage.
+
+## Report back
+Commit your work to the `ad2` branch and update ticket #32441 (hidden internal notes for engineering detail; the customer-facing thread is John Lehman). The remote operator will see your commits on sync. **Byte-for-byte DOS fidelity** (a leading `===` line + ~1-space input-column spacing, on ALL families) is intentionally deferred — there's a ticket note to disclose it to John when the whole effort is done.

projects/dataforth-dos/session-logs/2026-06/2026-06-17-mike-dataforth-datasheet-fixes.md

@@ -0,0 +1,87 @@
+## User
+- **User:** Mike Swanson (mike)
+- **Machine:** GURU-5070
+- **Role:** admin
+
+## Session Summary
+
+Drove the Dataforth test-datasheet pipeline remediation end-to-end on host AD2, opened from a `/sync` that surfaced an AD2 cloud session's diagnosis. AD2 (a fork that pushes to the `ad2` branch, NOT main) had already root-caused the datasheet defects John Lehman reported and pushed a diagnosis + proposals. Read John's two newest emails (which required repointing the broken `/mailbox` skill off a deleted Graph app), which added a live Phytec Germany deliverable and a concrete DSCA Defect-B example. Created Syncro ticket #32441 (Dataforth Corp, contact John Lehman, warranty labor) and emailed John a findings/plan summary.
+
+Ran a multi-AI (Grok adversarial + Gemini) review of the fix approaches, which materially hardened them, and committed the consolidated spec to main (`projects/dataforth-dos/DATASHEET-FIX-SPEC-2026-06-17.md`). Then, with backups in place (VSS shadow + 612 MB pg_dump) and per-file save-states, deployed three fixes to the live pipeline on AD2 via RMM (running node as SYSTEM): Fix 1 (RTD label), Fix 4 (encoded-serial importer recovery), and Fix 3 (retest "latest wins"). Each followed diagnose/validate-before-write with explicit verification.
+
+Fix 1: RTD modules rendered the input column as `Rin (ohms)` instead of `Temp. (C)`. Validated read-only (byte-compare vs staged originals; data correct, label fixed; remaining cosmetic diffs are pre-existing across all families — leading `===` line + ~1-space column shift — and were deferred per Mike). Deployed, re-pushed the 102 Phytec SCM5B35 certs (SO 120006-A) + the Wellbore audit unit 179553-13 to Hoffman (103 rows, 0 errors).
+
+Fix 4: importer dropped letter-prefixed encoded serials (DOS 8.3 encoding, e.g. `A243-1`=`10243-1`). A read-only pre-flight scan revealed the naive "decode any leading letter" produced false positives (lowercase/short tokens), so the decode rule was tightened to `^[A-Z]\d{3,}-`. Added `raw_serial_number` column + `test_records_quarantine` table; patched `multiline.js`+`import.js`; recovered 603 units (3 reused-serial collisions quarantined), published 518 to Hoffman.
+
+Fix 3: same-day retests could keep a non-final run. Added an `ingest_seq` tiebreaker (source mtime x 1e6 + line index); rewrote the conflict rule (date primary, same-day broken by ingest_seq, churn-guarded by `raw_data IS DISTINCT`, cross-model quarantined). SQL validated via a rolled-back execution test. The settle pass (re-import of ~9k multiline logs) was running in the background at save time.
+
+## Key Decisions
+- Filed the fix spec on `main` (not the `ad2` fork branch) and left AD2 a coord message; later switched to driving the fixes directly from GURU-5070 via RMM when Mike said VPN was up.
+- Multi-AI = the grok/gemini CLI skills (the established "MultiAI" pattern), not the Workflow tool.
+- Fix 2 (DSCA): per both AIs, DERIVE per-subtype templates from the staged `.TXT` originals (ground truth) rather than reverse-engineering the DOS `DSCFIN.DAT` binary. (Deferred — not yet built.)
+- Fix 4 decode rule restricted to `^[A-Z]\d{3,}-` after the pre-flight showed lowercase/short letter-prefixed serials are NOT 8.3 encodings (false positives). Kept the literal in `raw_serial_number`.
+- Fix 3 recency uses test_date primary with `ingest_seq` (mtime+line) only as a same-day tiebreaker — avoids the fragile pure-scan-order recency both AIs refuted, while keeping date (the reliable signal) authoritative.
+- Added a cross-model overwrite guard to the live `import.js` conflict rule (during Fix 4) so a different-model serial collision can never clobber an existing unit; full live-path quarantine routing remains Fix 3 territory.
+- Byte-for-byte DOS fidelity (the cosmetic render gap) deferred per Mike; recorded as a disclosure note to send John when the work is complete.
+- All production writes gated behind: VSS shadow + pg_dump backup, per-file timestamped `.bak` save-states, read-only validation first, atomic multi-patch (prepare-all-then-write), JS load-check, and (for Fix 3) a rolled-back SQL execution test.
+
+## Problems Encountered
+- `/mailbox` (Graph) was dead — `AADSTS700016`, the `Claude-MSP-Access` app `fabb3421` was deleted. The skill had been claimed-fixed in a prior session but never repointed. Repointed `mailbox.md` + memory to the dedicated mailbox app `1873b1b0` via `get-token.sh ... mailbox` (cert auth); logged a `--correction`.
+- Ticket `contact_id` would not set via POST or PUT (returned null) — but the Syncro GUI showed John as contact (API serialization quirk). Confirmed with Mike before sending the customer email.
+- First Phytec re-push matched only 58/102 — the DB stores a mix of padded (`179754-01`) and unpadded (`179754-1`) serials. Re-ran with both forms (union) → all 102.
+- WizTree zip (205 MB) was committed on the `ad2` branch. Moved it on AD2 into a new gitignored `clients/dataforth/local-artifacts/` dir + added the ignore rule, rather than racing AD2's active session with a competing commit.
+- RMM runs as SYSTEM; git balks at the repo on AD2 ("dubious ownership") — used `git -c safe.directory=...` for reads; did all DB/node work as SYSTEM (PG auth via db.js defaults works regardless of user).
+- `/tmp` path mismatch between Git Bash and Python recurred — passed tokens via env var instead of a `/tmp` file.
+
+## Configuration Changes
+Repo (main):
+- NEW `projects/dataforth-dos/DATASHEET-FIX-SPEC-2026-06-17.md` (commit f39516eb) — hardened multi-AI spec.
+- MODIFIED `.claude/commands/mailbox.md` — repointed off dead app `fabb3421` to mailbox app `1873b1b0` (get-token.sh mailbox tier); token() now delegates to get-token.sh.
+- MODIFIED `.claude/memory/feedback_365_remediation_tool.md` — fabb3421 marked DELETED; documented the mailbox app.
+- errorlog.md — correction (mailbox), plus earlier friction entries.
+
+AD2 production (`C:\Shares\testdatadb`, NOT in repo — deployed via RMM):
+- `templates/datasheet-exact.js` — RTD (sensorNum 7) folded into temperature path. Save-state `.bak-2026-06-17-1646`.
+- `parsers/multiline.js` — widened serial regex + decode + `raw_serial_number` (Fix 4); `fileMtime` + `ingest_seq` (Fix 3). Save-states `.bak-2026-06-17-1713`, `.bak-2026-06-17-1726`.
+- `database/import.js` — `raw_serial_number` in INSERT + cross-model guard (Fix 4); `ingest_seq` + new same-day conflict rule (Fix 3). Same save-states.
+- DB schema: `test_records.raw_serial_number` (TEXT), `test_records.ingest_seq` (BIGINT), NEW `test_records_quarantine` table.
+- NOTE: the repo copies under `projects/dataforth-dos/datasheet-pipeline/implementation/` are STALE vs deployed (e.g. repo import.js shows a 5-tuple ON CONFLICT; deployed uses serial_number). Always edit deployed; reconcile repo later.
+
+## Credentials & Secrets
+- Dataforth testdatadb PostgreSQL 18: host localhost (on AD2), port 5432, db `testdatadb`, user `testdatadb_app`, password `DfTestDB2026!` — found in plaintext in `database/db.js` defaults (no .env). VAULTED at `clients/dataforth/testdatadb-postgres.sops.yaml`.
+- Hoffman uploader OAuth creds at `C:\ProgramData\dataforth-uploader\credentials.json` on AD2 (ACL'd SYSTEM/Administrators/svc_testdatadb) — NOT vaulted (on-machine only); used by upload-to-api.js (client_credentials to CF_TOKEN_URL, bulk POST to CF_API_BASE/api/v1/TestReportDataFiles/bulk).
+- ACG mailbox Graph app (own tenant): `1873b1b0-3377-485c-a848-bae9b2f8f1f5`, vault `msp-tools/computerguru-mailbox.sops.yaml`, cert auth, SP disabled-when-idle.
+
+## Infrastructure & Servers
+- AD2 = 192.168.0.6, Windows Server 2019 Standard, RMM agent id `cfa93bb6-0cdc-4d4e-a29e-1609cda6f047` ("ACG Internal"). Test-datasheet pipeline host.
+- testdatadb: Node + PostgreSQL 18; long-running Windows service `testdatadb` (restart via `Restart-Service` or the `TestDataDB-ServiceRestart` scheduled task). Pipeline cron = scheduled task `DataforthTestDatasheetUploader` (run-pipeline.ps1, fresh node process each run). Other tasks: HoffmanInventoryPull, TestDataDB-Backup, TestDataDB-VacuumAnalyze.
+- Pipeline: DOS stations (.DAT logs, QuickBASIC) -> HISTLOGS (C:\Shares\test\Ate\HISTLOGS\<logtype>\<model>.DAT) + station logs (C:\Shares\test\TS-xx\LOGS) + Recovery (C:\Shares\Recovery-TEST) -> import.js -> test_records (474,383 rows after Fix 4) -> render-datasheet.renderContent -> upload-to-api -> Hoffman bulk API -> public TestDataReport.aspx.
+- Staged original datasheets (ground truth, pre-regeneration): C:\Shares\test\STAGE\<TS-station>\<encoded-SN>.TXT (8.3 hex-prefix encoding: 17->H, 18->I, etc.). 11,922 staged files.
+- pg_dump: C:\Program Files\PostgreSQL\18\bin\pg_dump.exe. Backups in C:\Shares\testdatadb\_backups\.
+- VSS active (153 GB shadow storage). C: ~362 GB free.
+
+## Commands & Outputs
+- Backup: VSS shadow `{fe3f53f6-e6b1-43f2-8da6-5be3b8cd2240}` (4:30 PM) + `testdatadb-2026-06-17-1630.dump` (611.9 MB).
+- Fix 1 validate: RTD scope = 23,937 rows (SCM5B34 9726, 8B35 5477, SCM5B35 5161, DSCA34 3573). Render byte-compare: data + final-test match; only cosmetic (leading `===` line + ~1-space column) differ (pre-existing, all families).
+- Phytec re-push: 103 rows (102 + audit unit), 45 updated + 58 unchanged, 0 errors.
+- Fix 4 pre-flight: 840 distinct dropped serials / 9,510 records / 100 models; 831 decoded-absent; 8 "collisions" = 5 false positives (lowercase/short) + 3 genuine (A819-1/A821-1/A821-2, reused across 8B34/8B36).
+- Fix 4 recovery: 606 genuine encoded units; 603 inserted, 3 quarantined, 0 errors; test_records 473,780 -> 474,383. Published: created=518, skipped=85 (no spec / unregistered model), 0 errors.
+- Fix 3 deploy: 7 patches OK, SQL-TEST valid (rolled-back upsert). Settle pass (re-import ~9k files) running at save time.
+
+## Pending / Incomplete Tasks
+- Fix 3 settle pass running (background) — when done: re-run same-day sweep (violations -> ~0), publish settled rows via uploadBySerialNumbers (controlled), document on #32441.
+- Fix 2 (DSCA Final-Test rebuild) — the big one; derive per-subtype templates from staged `.TXT`, byte-validate. NOT started.
+- Fix 5 (backfill 379 cryptolocker-era units) — publish staged `.TXT` directly via a `legacy_cert_text` column (decision pending). NOT started.
+- Bulk RTD re-push (~23,800 remaining RTD certs) — awaiting Mike's go.
+- 85 recovered units skipped at publish (no spec entry / unregistered Hoffman model) — investigate.
+- Byte-for-byte DOS fidelity — deferred; disclosure note to John recorded on ticket (comment 419546930) to send when done.
+- Reconcile the stale repo copies under `projects/dataforth-dos/datasheet-pipeline/implementation/` with the deployed files.
+- coord API not reachable from AD2 (no VPN there) — Mike noted "make coord VPN-optional" as a future idea.
+
+## Reference Information
+- Syncro ticket #32441 (id 112783550), Dataforth Corp (cust 578095, prepay 31.5h), contact John Lehman (2851723, jlehman@dataforth.com). Comments: Initial Issue 419541902, internal accounting 419541903, findings email 419542053, byte-fidelity follow-up 419546930, deploy record 419547387, progress update (emailed) 419548057, Fix-4 record 419549891.
+- Spec: projects/dataforth-dos/DATASHEET-FIX-SPEC-2026-06-17.md (commit f39516eb).
+- AD2 ad2-branch docs: DATASHEET-RTD-BUG-DIAGNOSIS / PARSING-FIDELITY-VERDICT / MISSING-UNITS-REPORT-FOR-JOHN / CONFLICT-RULE-FIX-PROPOSAL / EMAIL-TO-JOHN-datasheet-findings (all 2026-06-17).
+- Phytec SO 120006-A missing serials (now published): SCM5B35-02D 179754-01..32, 179756-01..07/09..21; SCM5B35-01D 178413-06..16, 179736-01..05; SCM5B35-1761 179740-01..13, 179753-01..21.
+- AD2 save-states: datasheet-exact.js.bak-2026-06-17-1646; multiline.js/import.js .bak-2026-06-17-1713 (Fix4) and .bak-2026-06-17-1726 (Fix3).
+- Encoded-serial decode: leading uppercase letter L, real prefix = String(L.charCodeAt(0)-55) (A=10..H=17..I=18); only applied to `^[A-Z]\d{3,}-`.

projects/dataforth-dos/tools/derive-dsca-templates.js

@@ -0,0 +1,53 @@
+// Fix 2 STAGE 1 (read-only build): extract per-model DSCA Final-Test templates from staged originals.
+// Uses the '===' separator line under the Final-Test header to get exact column spans.
+const fs = require('fs'), path = require('path');
+const STAGE = 'C:/Shares/test/STAGE';
+const OUT = 'C:/Shares/testdatadb/dsca-templates.json';
+function walk(d, out) { let it = []; try { it = fs.readdirSync(d, { withFileTypes: true }); } catch { return out; } for (const e of it) { const p = path.join(d, e.name); if (e.isDirectory()) walk(p, out); else if (/\.txt$/i.test(e.name)) out.push(p); } return out; }
+function colSpans(sep) { const cols = []; let m; const re = /=+/g; while ((m = re.exec(sep))) cols.push([m.index, m.index + m[0].length]); return cols; }
+function extract(t) {
+  const lines = t.replace(/\r\n/g, '\n').split('\n');
+  const accHdr = lines.find(l => /Error \(%\)/.test(l) && /Status/.test(l)) || '';
+  const accOut = (accHdr.match(/Output \((?:V|mA)\)|Vout \(V\)/) || ['?'])[0];
+  let fi = lines.findIndex(l => /FINAL TEST RESULTS/.test(l)); if (fi < 0) return null;
+  let hi = -1; for (let i = fi + 1; i < lines.length; i++) { if (/Parameter\s+Measured/.test(lines[i])) { hi = i; break; } } if (hi < 0) return null;
+  const sep = lines[hi + 1] || ''; if (!/=/.test(sep)) return null;
+  const cols = colSpans(sep); if (cols.length < 4) return null;
+  const [pc, mc, sc, stc] = cols;
+  const rows = [];
+  for (let i = hi + 2; i < lines.length; i++) {
+    const l = lines[i];
+    if (/Check List|^\s*_{5,}/.test(l)) break;
+    if (!l.trim()) continue;
+    const name = (l.slice(pc[0], mc[0]) || '').trim();
+    const spec = (l.slice(sc[0], stc[0]) || '').trim();
+    if (!name && !spec) continue;
+    rows.push({ name, spec });
+  }
+  return { accOut, rows };
+}
+(async () => {
+  const files = walk(STAGE, []);
+  const byModel = {};
+  for (const f of files) {
+    let t; try { t = fs.readFileSync(f, 'utf8'); } catch { continue; }
+    const model = (t.match(/^\s*Model:\s*(\S+)/m) || [])[1] || '';
+    if (!/^DSCA/i.test(model)) continue;
+    const tpl = extract(t); if (!tpl) continue;
+    // keep the sheet with the MOST rows per model (most complete; avoids truncated samples)
+    if (!byModel[model] || tpl.rows.length > byModel[model].rows.length) byModel[model] = { ...tpl, sheets: (byModel[model] ? byModel[model].sheets : 0) + 1 };
+    else byModel[model].sheets++;
+  }
+  const models = Object.keys(byModel).sort();
+  console.log('DSCA models templated: ' + models.length);
+  const out = {}; for (const m of models) out[m] = { accOut: byModel[m].accOut, rows: byModel[m].rows };
+  fs.writeFileSync(OUT, JSON.stringify(out));
+  console.log('wrote ' + OUT + ' (' + fs.statSync(OUT).size + ' bytes)');
+  const rc = {}; for (const m of models) { const n = byModel[m].rows.length; rc[n] = (rc[n] || 0) + 1; }
+  console.log('row-count distribution (rows:models): ' + Object.entries(rc).sort((a, b) => a[0] - b[0]).map(([n, c]) => n + ':' + c).join('  '));
+  for (const probe of ['DSCA38-05', 'DSCA34-01', 'DSCA38-08C', 'DSCA30-01']) {
+    const s = out[probe];
+    if (s) { console.log('\n' + probe + '  accOut=' + s.accOut + '  rows=' + s.rows.length); s.rows.forEach(r => console.log('   ' + r.name.padEnd(28) + ' | ' + r.spec)); }
+    else console.log('\n' + probe + ': NOT FOUND');
+  }
+})().catch(e => { console.error('ERR ' + e.message); });