sync: auto-sync from GURU-5070 at 2026-06-12 05:57:38

Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-06-12 05:57:38

.claude/commands/rmm.md

@@ -157,6 +157,8 @@ Show: hostname, os_type, online/offline, client_name (from `site_name`/`client_n
 
 Use `python` only when explicitly writing a Python script. Use `script` for saved scripts (not covered in this skill).
 
+**VALID `command_type` values ONLY: `shell`, `powershell`, `python`, `script`, `claude_task` (plus alias `cmd` → shell = cmd.exe).** The agent deserializes `command_type` into a Rust enum; an UNKNOWN value (e.g. a made-up type) fails the agent's whole-message JSON parse and the command is **silently dropped — no ack, no result, no error** — which is indistinguishable from a network black-hole and has caused a long mis-diagnosis. On Windows: `powershell` runs powershell.exe (UTF-8 output fixed in-agent); `shell` or `cmd` runs cmd.exe. If a dispatched command sits un-acked forever, FIRST suspect an invalid `command_type` before chasing the network. (Newer agents NAK an unparseable command so it fails fast with a clear stderr instead of black-holing.)
+
 ### Basic dispatch
 
 ```bash

.claude/memory/MEMORY.md

@@ -132,3 +132,4 @@
 - [gururmm session-logs are in a submodule](gururmm-session-logs-submodule-save.md) — commit in the submodule + `git push origin HEAD:main` (GURU-5070 CAN push over HTTP now); then advance the parent gitlink
 - [Use `python` not `python3` on GURU-5070](python3-shim-use-python.md) — `python3` in Git bash hits the flaky MS Store shim; real interpreters are `python` (3.12) / `py` (3.14). coord.py + wiki-compile work via `python`; the coord lock IS claimable here
 - [Beast = primary GuruRMM Windows build host](gururmm-beast-windows-build-host.md) — GURU-BEAST-ROG (i9), reached from .30 via Tailscale-on-.30 at 100.101.122.4 as guru; Pluto is the fallback (`attempt_build beast || attempt_build pluto`). WiX must be 4.x (v6+ = OSMF); Beast NuGet needed nuget.org added
+- [GuruRMM command_type gotcha](reference_gururmm_command_type.md) — only shell/powershell/python/script/claude_task (+cmd alias); unknown type silently dropped, looks like a black-hole

.claude/memory/reference_gururmm_command_type.md

@@ -0,0 +1,25 @@
+---
+name: reference_gururmm_command_type
+description: GuruRMM agent only accepts specific command_type values; an unknown type is silently dropped (looks like a black-hole)
+metadata:
+  type: reference
+---
+
+GuruRMM agent `CommandType` (agent/src/transport/mod.rs) accepts ONLY: `shell`,
+`powershell`, `python`, `script`, `claude_task` — plus alias `cmd` → shell
+(added 2026-06-12). On Windows: `powershell` runs powershell.exe (UTF-8 output
+fixed in-agent, so the old "-OutputEncoding not recognized" quirk is gone);
+`shell`/`cmd` runs cmd.exe.
+
+A command with an UNKNOWN `command_type` fails the agent's whole-message serde
+parse; pre-2026-06-12 the error was logged-and-ignored and the command was
+**silently dropped — no ack, no result** — indistinguishable from a NAT/proxy
+black-hole. On 2026-06-12 a `command_type:"cmd"` (no variant then) caused a long
+mis-diagnosis (7 multi-AI rounds, packet captures, a pfSense SNAT change) of
+"PST agents can't receive commands" — the agents ran `powershell` commands fine
+the whole time. The agent now also NAKs an unparseable command (CommandAck +
+error CommandResult) so it fails fast instead of black-holing.
+
+**How to apply:** When a dispatched command sits un-acked/never-completes,
+FIRST verify `command_type` is one of the valid values before chasing the
+network/proxy. Never send a made-up type. See [[reference_gururmm]].

clients/sheldon-goldstein/session-logs/2026-06/2026-06-12-mike-goldstein-vpn-reset-billing.md

@@ -0,0 +1,60 @@
+## User
+- **User:** Mike Swanson (mike)
+- **Machine:** GURU-5070
+- **Role:** admin
+
+## Session Summary
+
+Billed a 30-minute remote labor charge for resetting the VPN on the Tucson UCG-Ultra (UniFi Cloud Gateway Ultra) gateway for Sheldon Goldstein. The request was to find "today's" VPN-related ticket and log the work, but a search of both Goldstein Syncro accounts (Sheldon Goldstein cust 25307933, and Goldstein & Scopellite PC cust 852314) found no ticket created on 2026-06-11. The nearest VPN/remote-access ticket was #32384 ("Remote - Issues with remote access"), created 2026-06-04 and already in Invoiced status.
+
+Per Mike's decision, the new work was added to existing ticket #32384 rather than creating a fresh ticket. Because #32384 was already invoiced (Winter's prior 1.0h remote line, $150, invoiced 6/5), the ticket was reopened to In Progress, a customer-visible resolution comment was added, and a new 0.5h remote labor line ($75, attributed to Mike) was billed. A new invoice (#67820) was created and verified to contain only the new $75 line — the original $150 invoice was not re-billed, confirming no double-billing.
+
+The ticket was then set back to Invoiced, and finally Resolved when Mike asked to close it (Resolved is the terminal "closed" state in ACG's Syncro workflow). Bot alerts were posted to #bot-alerts after each write (bill, resolve).
+
+## Key Decisions
+
+- Added today's work to existing ticket #32384 instead of opening a new ticket (Mike's call), accepting the reopen-an-invoiced-ticket workflow.
+- Verified the second invoice captured only the new unbilled $75 line before proceeding — guard against Syncro re-pulling the already-invoiced $150 line. Confirmed total = $75.00.
+- New labor line attributed to Mike (user_id 1735) since Mike performed today's work; the original line stays attributed to Winter (user_id 1737).
+- Standard remote rate, non-prepaid (customer prepay_hours = 0.0): 0.5h x $150 = $75.
+
+## Problems Encountered
+
+- No Goldstein ticket existed for 2026-06-11 despite the request to "find today's ticket." Resolved by surfacing the gap and letting Mike choose the target ticket (#32384).
+- #32384 was already Invoiced, so a naive re-invoice risked double-billing the original $150 line. Mitigated by reopening, adding only the new line, then verifying the new invoice total = $75.00 (only the new line present).
+
+## Configuration Changes
+
+None (no repo files modified; Syncro-only work).
+
+## Credentials & Secrets
+
+None newly discovered or created. Syncro per-user API keys are baked into the syncro skill.
+
+## Infrastructure & Servers
+
+- Device worked on: Tucson UCG-Ultra (UniFi Cloud Gateway Ultra) at Sheldon Goldstein. VPN was reset; remote access confirmed restored.
+
+## Commands & Outputs
+
+- Customer search `goldstein` -> Sheldon Goldstein (25307933), Goldstein & Scopellite PC (852314).
+- `GET /tickets?customer_id=...` -> no ticket dated 2026-06-11; #32384 (Invoiced, 2026-06-04) is nearest VPN/remote-access ticket.
+- Reopen: `PUT /tickets/112201728 {"status":"In Progress"}` -> In Progress.
+- Comment id 418639597 posted (customer-visible resolution).
+- `POST /tickets/112201728/add_line_item` -> line id 42832094, 0.5h @ $150, user_id 1735.
+- `POST /invoices` -> invoice #67820 (id 1650653134), total $75.00, single line only.
+- `PUT /tickets/112201728 {"status":"Invoiced"}` then `{"status":"Resolved"}`.
+
+## Pending / Incomplete Tasks
+
+None. Work billed, invoiced, and ticket resolved.
+
+## Reference Information
+
+- Customer: Sheldon Goldstein, cust_id 25307933, email sheldon@lawyersdallas.com
+- Ticket #32384, internal id 112201728 — https://computerguru.syncromsp.com/tickets/112201728
+- Original line: id 42749720, Winter (1737), 1.0h remote, $150 (invoiced 6/5)
+- New line: id 42832094, Mike (1735), 0.5h remote, $75
+- New invoice: #67820, id 1650653134, $75.00
+- Remote labor product: 1190473 (Labor - Remote Business), $150/hr
+- Bot alerts: message_id 1514803261935325344 (billed), 1514804079191265351 (resolved)

tmp/addnpmnat.php

@@ -0,0 +1,24 @@
+<?php
+require_once("config.inc");
+require_once("filter.inc");
+global $config;
+init_config_arr(array('nat','outbound','rule'));
+$rules =& $config['nat']['outbound']['rule'];
+foreach ($rules as $r) {
+  if (($r['source']['network'] ?? '') === '172.16.3.20/32' && ($r['target'] ?? '') === '72.194.62.10') { echo "ALREADY_EXISTS\n"; exit(0); }
+}
+$new = array(
+  'source' => array('network' => '172.16.3.20/32'),
+  'sourceport' => '',
+  'descr' => 'NPM rmm-api server->agent reply source pin -> 72.194.62.10 (RMM cmd-delivery diag)',
+  'target' => '72.194.62.10',
+  'target_subnet' => '',
+  'interface' => 'wan',
+  'poolopts' => '',
+  'source_hash_key' => '',
+  'destination' => array('any' => ''),
+);
+array_unshift($rules, $new);
+write_config("Add NPM (172.16.3.20) outbound SNAT pin -> 72.194.62.10");
+$ret = filter_configure();
+echo "ADDED ret=$ret total_outbound_rules=".count($rules)."\n";

tmp/delnpmnat.php

@@ -0,0 +1,14 @@
+<?php
+require_once("config.inc");
+require_once("filter.inc");
+global $config;
+init_config_arr(array('nat','outbound','rule'));
+$rules =& $config['nat']['outbound']['rule'];
+$before = count($rules);
+$rules = array_values(array_filter($rules, function($r){
+  return !((($r['source']['network'] ?? '') === '172.16.3.20/32') && (($r['target'] ?? '') === '72.194.62.10'));
+}));
+$config['nat']['outbound']['rule'] = $rules;
+write_config("Roll back NPM SNAT pin (diagnostic, removed)");
+filter_configure();
+echo "removed=".($before-count($rules))." remaining_outbound=".count($rules)."\n";

tmp/jssh.py

@@ -0,0 +1,15 @@
+#!/usr/bin/env python3
+"""Run a command on Jupiter (172.16.3.20) via password SSH. Usage: jssh.py '<cmd>'"""
+import sys, paramiko
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("172.16.3.20", port=22, username="root", password="Th1nk3r^99##",
+          look_for_keys=False, allow_agent=False, timeout=20)
+cmd = sys.argv[1]
+stdin, stdout, stderr = c.exec_command(cmd, timeout=int(sys.argv[2]) if len(sys.argv) > 2 else 60)
+out = stdout.read().decode(errors="replace")
+err = stderr.read().decode(errors="replace")
+sys.stdout.write(out)
+if err.strip():
+    sys.stderr.write("\n--STDERR--\n" + err)
+c.close()

tmp/ns-cap.out

@@ -0,0 +1,23 @@
+Traceback (most recent call last):
+  File "C:\Users\guru\AppData\Local\Programs\Python\Python312\Lib\site-packages\paramiko\channel.py", line 697, in recv
+    out = self.in_buffer.read(nbytes, self.timeout)
+          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "C:\Users\guru\AppData\Local\Programs\Python\Python312\Lib\site-packages\paramiko\buffered_pipe.py", line 154, in read
+    raise PipeTimeout()
+paramiko.buffered_pipe.PipeTimeout
+
+During handling of the above exception, another exception occurred:
+
+Traceback (most recent call last):
+  File "D:\ClaudeTools\tmp\jssh.py", line 10, in <module>
+    out = stdout.read().decode(errors="replace")
+          ^^^^^^^^^^^^^
+  File "C:\Users\guru\AppData\Local\Programs\Python\Python312\Lib\site-packages\paramiko\file.py", line 184, in read
+    new_data = self._read(self._DEFAULT_BUFSIZE)
+               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "C:\Users\guru\AppData\Local\Programs\Python\Python312\Lib\site-packages\paramiko\channel.py", line 1359, in _read
+    return self.channel.recv(size)
+           ^^^^^^^^^^^^^^^^^^^^^^^
+  File "C:\Users\guru\AppData\Local\Programs\Python\Python312\Lib\site-packages\paramiko\channel.py", line 699, in recv
+    raise socket.timeout()
+TimeoutError

tmp/ns3.out

@@ -0,0 +1 @@
+DONE

tmp/r1-gemini.txt

@@ -0,0 +1,34 @@
+# GuruRMM — agent command delivery failure: diagnostic dossier (neutral)
+
+The evidence suggests a highly selective delivery failure where **outbound (Server -> Agent) WebSocket Text frames** are being dropped, mangled, or ignored, while **WebSocket Control frames (Ping)** and **inbound (Agent -> Server) traffic** remain functional.
+
+### 1. Candidate Root-Cause Hypotheses
+
+| Hypothesis | Reasoning / Evidence FOR | Reasoning / Evidence AGAINST |
+| :--- | :--- | :--- |
+| **H1: MTU / Fragmentation (Large Frame Blackhole)** | Common in site-to-site tunnels (UniFi UDR). Large commands (4KB) + overhead might exceed path MTU, causing fragmentation. If the path drops fragments, the Text frame never arrives. | **Refuted by T2/T3/T9:** Tiny `hostname` commands (~80B) also failed. MTU issues usually spare small packets. |
+| **H2: Application-Layer Deadlock (Agent Rx Loop)** | If the Agent's `handle_server_message` or the internal `tx` (to the executor) blocks or panics without crashing the read loop, the agent stops processing Text frames but continues to receive Pings (keeping the 90s deadline alive). | **Weakened by T7/T8:** A server-forced eviction and fresh reconnect (fresh Agent state) did not restore functionality. |
+| **H3: Intermediate Proxy / Firewall (Text-Frame Filtering)** | Cloudflare or a local "Security Gateway" (UniFi) may be inspecting WebSocket payloads. If the Command JSON pattern matches a signature or if the `Upgrade` handshake is being "half-policed," Text frames might be dropped. | The fleet baseline (40 agents working) suggests it's not a global Cloudflare issue, but could be specific to "enhanced" security profiles at certain sites. |
+| **H4: Protocol Desync (ConfigUpdate Serialization)** | On connect, the Server sends a `ConfigUpdate` (multi-KB). If this frame is malformed or if `tokio-tungstenite` on the server-side has a buffer issue with specific frame sequences, the Agent's parser might be stuck/errored for that specific stream. | **Supported by T8:** T8 confirms that even after a reconnect, delivery fails. This points to a failure during or immediately after the handshake/ConfigUpdate sequence. |
+| **H5: Nginx/Proxy Buffer "Clogging"** | `proxy_read_timeout` is long, but `proxy_buffers` or internal memory limits on the origin/nginx could be hit for specific connections if the socket becomes "write-blocked" from the origin to Nginx. | **Against:** Nginx logs would typically show 499s or buffer overflows; `send_to` on the server returns `success`, meaning it's reaching the OS/loopback buffer. |
+
+---
+
+### 2. The Verdict on the Evidence
+
+*   **Inference Check:** The claim that "the agent was receiving SOME inbound frames" (T4) is an **unverified inference**. While the 90s timeout didn't trigger, we cannot distinguish between "the agent received a Ping" and "the agent's internal timer is somehow stalled/incorrect."
+*   **Contradiction:** T8 is the most troubling. If a fresh connection (after eviction) doesn't fix it, the issue is either **environmental (Network/Path)** or **intrinsic to the Agent's 0.6.63 startup logic** (e.g., it crashes its own Text handler immediately upon receiving the first `ConfigUpdate` or `Command`).
+
+---
+
+### 3. The SINGLE Most Diagnostic Test
+
+**Action:** Dispatch a **WebSocket Ping from the Server** to a stuck agent and **explicitly log the receipt of the corresponding Pong**.
+
+**Reasoning:**
+*   **If Pong is received:** We confirm the path is fully transparent to WebSocket *Control* frames. This isolates the failure to **Text-frame serialization, size, or content inspection** (H3/H4).
+*   **If Pong is NOT received:** The agent is "ghosting" the server entirely. This means the 90s timeout in the agent (T4) is the actual failure point — the agent *should* have reconnected but didn't. This points to an **Agent-side Rx loop deadlock** (H2).
+
+**Outcome-based path:**
+*   **Pong Received:** Inspect Nginx/Cloudflare for "Web Application Firewall" (WAF) drops or "WebSocket inspection" settings.
+*   **Pong NOT Received:** The agent is dead-locked. Investigate the `tokio` task management in the Agent's 0.6.63 read loop.

tmp/r1-grok.txt

@@ -0,0 +1,139 @@
+# Diagnostic analysis: GuruRMM command delivery failure
+
+## What the evidence actually proves
+
+**Established facts:**
+- Agent → server path is healthy for affected agents: `last_seen` stays current, Heartbeats arrive, re-offer logic fires on Heartbeat.
+- Server believes it is dispatching: `send_to()` returns true, reaper logs re-delivery, no enqueue failures.
+- Affected agents never send `CommandAck` and never execute commands (large and tiny).
+- Most of the fleet (~40 agents, 397 commands in 6h) is unaffected on the same server build.
+- PST-SERVER had one successful acked command at 22:23:40 on 0.6.63; failure begins after the 00:22:03 reconnect.
+- Eviction → reconnect works mechanically on the server side, but commands still fail on the fresh connection (T8).
+
+**Unverified inferences (treat as hypotheses, not facts):**
+- “Agent is receiving server Pings” — inferred only from *no* 90s agent-side reconnect (T4). No Pong logging on the server; server code ignores inbound `Pong`.
+- “`send_to` success means the command was sent” — it only means the in-memory channel accepted the message.
+- “ConfigUpdate was delivered on reconnect” — server logged `Sent ConfigUpdate`; no agent-side confirmation.
+- PST-SERVER, PST-SERVER2, and GTS-PEDRO-H share one root cause — GTS failed with `attempts=1` at a 60s timeout, not the 10-attempt undeliverable pattern; it had been idle since 06-06. That may be a second failure mode.
+
+**The core puzzle:** This is not a dead connection in the usual sense. It looks like a **live, asymmetric channel**: inbound app traffic works; outbound *command delivery* does not — yet the connection is stable enough that the agent does not trigger its 90s “no inbound activity” reconnect.
+
+---
+
+## Ranked root-cause hypotheses
+
+### 1. Server outbound Text writes succeed at enqueue but fail (or stall) at the socket — while Ping writes still work via `select!` interleaving
+**For:**
+- `send_to()` only checks `tx.send()` on an (likely unbounded) channel, not socket delivery.
+- Send task uses one loop for Text and Ping; a **stuck `sender.send(Text).await`** does not necessarily kill the task immediately; Ping ticks can still fire when not blocked inside a Text send.
+- Would explain: heartbeats in, no CommandAck, no execution, reaper climbing attempts, “online” agent.
+- Fresh reconnect (T8) could still fail if the underlying TCP/WebSocket path is half-open for application Text but still passes occasional Ping frames.
+
+**Against:**
+- If the send task were **permanently** blocked on Text, Pings would stop and the agent should reconnect every ~90s. T4 shows a 30+ minute stable connection — so either Pings are getting through, or the 90s inference is wrong.
+- After eviction, a brand-new TCP session should reset middlebox state; T8 still fails, which weakens “stuck write on old socket” unless the pathology is immediate on every new session at that site.
+
+**Verdict:** Strong, but needs socket-level write confirmation. The contradiction around 90s stability is the main gap.
+
+---
+
+### 2. Agent receives inbound frames (at least Pings) but server → agent **Text** frames are dropped or never parsed — agent-side or on-path selective loss
+**For:**
+- Best fit for “connection alive, heartbeats out, commands never acked/executed.”
+- Agent ACKs on receipt *before* execution; no ACK + no execution ⇒ Command handler likely never ran on a parsed `Command`.
+- Tiny `hostname` (~80 B) fails (T2, T3, T8) — rules out payload size as the primary filter.
+- PST-SERVER regressed exactly at reconnect — first thing on a new session is `AuthAck`, `ConfigUpdate`, then command re-offers. We do not know whether *any* post-reconnect Text reached the agent.
+
+**Against:**
+- WebSocket middleboxes rarely drop only `Text` opcode frames while passing `Ping`. Possible but unusual unless something is inspecting JSON content.
+- Does not by itself explain GTS-PEDRO-H at a different site unless this is a broader server bug in Text serialization for certain agents.
+
+**Verdict:** Very plausible. “Ping works, Text doesn’t” is an inference, not a measurement — but the symptom shape matches.
+
+---
+
+### 3. Agent-side bug or state corruption in 0.6.63 (Command path broken; Heartbeat/metrics path fine)
+**For:**
+- PST-SERVER2 has **never** acked a command on 0.6.63 — suggests a persistent local pathology, not a transient network blip.
+- PST-SERVER worked once, then failed after reconnect — consistent with handshake/`ConfigUpdate` leaving the command handler or ACK channel in a bad state.
+- Agent architecture likely separates heartbeat/metrics tasks from the read/command path; one can work while the other is broken.
+- Explains T8: new server connection does not help if the agent process state is wrong.
+
+**Against:**
+- Hard to explain why 40 other agents on 0.6.63 are fine unless trigger is site-specific policy/config in `ConfigUpdate`.
+- Without agent logs, this is hard to distinguish from hypothesis #2.
+
+**Verdict:** High plausibility, especially for PST-SERVER2. May combine with #2 (bad config triggers bug).
+
+---
+
+### 4. Site/gateway pathology (UniFi UDR “Cloudflare-Ultra” class) affecting server → client WebSocket application data
+**For:**
+- PST-SERVER and PST-SERVER2 share site and gateway; both broken, overlapping timeframe.
+- UniFi + aggressive DNS/CF integration could plausibly affect long-lived WSS in non-obvious ways.
+
+**Against:**
+- GTS-PEDRO-H is a different site and also fails — unless that’s unrelated (see #6).
+- Fleet-wide Cloudflare + nginx path works for 40 agents; edge config would need to be connection- or path-specific.
+- Tiny frames also fail — less consistent with typical MTU/DPI size limits.
+
+**Verdict:** Plausible for PST pair alone; weak as a single explanation for all three agents.
+
+---
+
+### 5. Stale/wrong connection routing in server in-memory agent map (commands enqueued to a dead sender; heartbeats handled elsewhere)
+**For:**
+- Classic split-brain pattern for connection registries.
+- `send_to` returning true on a queued-but-never-drained channel matches symptoms.
+
+**Against:**
+- Eviction explicitly closed the connection and re-ran handshake (T7); map should have been replaced.
+- Single nginx → single Rust server process; no load-balancer ambiguity described.
+- If receive task is dead, `last_seen` would not update — it does.
+
+**Verdict:** Lower plausibility after T7/T8, unless there is a subtle duplicate-registration bug not ruled out by eviction.
+
+---
+
+### 6. Multiple independent failure modes grouped by coincidence
+**For:**
+- GTS-PEDRO-H: 6-day command gap, `attempts=1`, 60s timeout — different from PST’s climb to 10 “undeliverable.”
+- Pre-06-12 successes with `acked=false` are expected (pre-CommandAck).
+- One test on GTS (T9) is thin evidence to merge it with PST.
+
+**Against:**
+- All three are 0.6.63 and fail the same operator test (`hostname`, no ack).
+
+**Verdict:** Do not assume one root cause for all three without more data. GTS may be a red herring or a second bug.
+
+---
+
+### 7. Command JSON / server message construction bug affecting only certain agents
+**For:**
+- Could cause agent to silently drop unparsable `Text` while still handling other message types — if those other types were received, which we have not confirmed post-reconnect.
+
+**Against:**
+- Same server code path completes 397 commands for other agents.
+- Would need agent-specific command content or policy embedding in the `Command` frame.
+
+**Verdict:** Low unless `ConfigUpdate` policy differs per agent and corrupts something downstream.
+
+---
+
+## Single most diagnostic next step
+
+**Instrument the server per-connection send task and receive loop for one affected agent (PST-SERVER): log (with monotonic timestamps and `connection_id`) every completed `sender.send()` by frame type (`Text` vs `Ping`), Text byte length and `command_id`, and every inbound `Pong` correlated to the preceding `Ping`; then dispatch one tiny `hostname` command.**
+
+No agent access required. Run for 2–3 minutes on the stuck agent.
+
+| Outcome | Interpretation |
+|--------|----------------|
+| **Ping writes succeed + inbound Pongs observed; Text write for `hostname` logs success; still no `CommandAck`** | Server → socket → nginx path delivers control frames; **agent is not processing server Text** (agent bug, or on-path Text opcode loss after origin). Strongest signal for #2 / #3. |
+| **Ping writes succeed + Pongs observed; Text write errors or never completes** | **Server-side or loopback socket pathology** on application Text (#1). |
+| **Ping writes logged but no inbound Pongs** | The “agent receives Pings” inference was wrong; connection is stranger than modeled — revisit half-open TCP, nginx connection mapping, or agent timeout logic (#1 / #5). |
+| **Neither Ping writes nor Pongs, yet agent stays “online” and heartbeats continue** | **Split connection or duplicate session**: inbound and outbound are not the same WebSocket the send task thinks (#5). Escalate to connection registry + nginx upstream connection audit. |
+| **Text write succeeds on fresh post-eviction `connection_id`, same as above** | Rules out “stale socket only”; points to **persistent agent-side or site-path Text loss** (#2/#3/#4). |
+
+That one test separates “server thinks it sent” from “bytes left the process,” validates or falsifies the Ping inference, and narrows the fault domain to **origin socket / agent / middlebox** without needing endpoint access first.
+
+**Secondary high-value data (if the above is already partially instrumented):** nginx `tcpdump` or OpenResty debug on loopback `:3001` during a `hostname` dispatch — confirms whether Text frames leave the Rust process into nginx. That does not prove WAN delivery, but it collapses #1 vs #2/#3 on the origin host in one capture.

tmp/r2-gemini-prompt.md

@@ -0,0 +1,117 @@
+# GuruRMM command-delivery diagnosis — ROUND 2 (important topology correction)
+
+You analyzed this in round 1 (your prior answer is included at the bottom). A FACTUAL CORRECTION
+to the architecture has come to light that invalidates part of the round-1 premise. Please
+re-evaluate from scratch where needed. Be willing to discard your prior top hypothesis.
+
+## CRITICAL CORRECTION to the ingress topology
+
+In round 1 we told you: "agent -> Cloudflare -> nginx -> Rust server". **That was wrong for the
+AGENT path.** Verified facts now:
+
+- The agent's hard-coded WebSocket URL is **`wss://rmm-api.azcomputerguru.com/ws`** (from agent source:
+  `DEFAULT_SERVER_URL`, and the agent config default). The installer and enrollment also use
+  `rmm-api.azcomputerguru.com`.
+- DNS / Cloudflare proxy status for the two hostnames (both A-record to the same public IP 72.194.62.10):
+  - `rmm.azcomputerguru.com`     -> **proxied = true**  (orange cloud; goes THROUGH Cloudflare)  — this is the human DASHBOARD.
+  - `rmm-api.azcomputerguru.com` -> **proxied = false** (grey cloud; **DNS-only, BYPASSES Cloudflare**) — this is what the AGENTS use.
+- Therefore **Cloudflare is NOT in the agent's path at all.** All round-1 hypotheses about Cloudflare
+  WAF / CF WebSocket buffering / CF frame handling are moot for agent command delivery. (Cloudflare only
+  fronts the dashboard.)
+
+## The ACTUAL agent path (verified)
+
+```
+agent (endpoint, e.g. PST-SERVER behind a UniFi UDR/"Cloudflare-Ultra"-class gateway)
+  -> endpoint LAN/NAT
+  -> public internet
+  -> 72.194.62.10  (public IP; this is the NPM box)
+  -> NPM = "Nginx Proxy Manager" on host 172.16.3.20  (terminates TLS; one nginx layer)
+       NPM proxy-host settings for rmm-api: forward http://172.16.3.30:80, websockets=ON,
+       http2_support=ON, block_exploits=ON (NPM "Block Common Exploits"), caching=OFF, advanced_config=EMPTY
+  -> http://172.16.3.30:80   (the ORIGIN nginx; a SECOND nginx layer)  -- PLAINTEXT HTTP over the LAN here
+  -> proxy_pass http://127.0.0.1:3001   (the Rust server)
+```
+
+So there are **TWO nginx proxy layers** in series (NPM on .20, then origin nginx on .30), no CDN.
+
+Origin nginx `/ws` block (verbatim):
+```
+location /ws {
+    proxy_pass http://127.0.0.1:3001;
+    proxy_http_version 1.1;
+    proxy_set_header Upgrade $http_upgrade;
+    proxy_set_header Connection "upgrade";
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_read_timeout 86400;
+}
+```
+(No explicit `proxy_buffering off;`, no `proxy_send_timeout`. NPM's generated config for the proxy host is
+the standard NPM template with websockets enabled + block_exploits; we have not yet dumped its exact directives.)
+
+## What this changes / things to reconsider
+
+- The "asymmetry" symptom is unchanged and still the core puzzle: for affected agents the server keeps
+  receiving inbound (heartbeats, last_seen fresh), the agent does NOT trip its 90s no-inbound reconnect
+  (so it appears to be receiving at least the server's 30s WS Ping frames), yet server->agent COMMAND (Text)
+  frames are never acked/executed — both ~4 KB commands AND ~80 B `hostname` commands. One command DID
+  succeed earlier (right after the agent updated/reconnected). A server-forced eviction + fresh reconnect did
+  NOT restore command delivery (a tiny command on the fresh connection still never acked) — but note we have
+  NO confirmation the agent received anything server->agent on that fresh connection either.
+- Affected: PST-SERVER + PST-SERVER2 (same site/UDR), and GTS-PEDRO-H (different site). ~40 other agents
+  deliver commands fine through this SAME NPM+nginx path.
+- Because rmm-api is grey-cloud, the agent's TLS connection goes straight from the endpoint's gateway to the
+  public IP — there is no CDN normalizing MTU/keepalive/buffering on the path; the endpoint's own gateway
+  (e.g. PST's UDR) is directly in the path.
+
+## A note on one earlier "test"
+
+We tried to `tcpdump` the origin loopback (nginx->Rust on 127.0.0.1:3001) during a marked command dispatch and
+saw 0 packets — but that capture almost certainly FAILED (the backgrounded tcpdump was killed when the SSH
+session closed), so treat it as NO DATA, not as evidence that the loopback is silent.
+
+## Please now provide (round 2)
+
+1. Your REVISED ranked hypotheses given Cloudflare is out of the agent path and there are two nginx layers +
+   the endpoint gateway directly in the path. Explicitly say which round-1 hypotheses you drop or demote.
+2. The single most diagnostic next test (we can run packet captures on the .30 host on the plaintext .20->.30:80
+   leg and on the 127.0.0.1:3001 loopback; we can read NPM's generated nginx config on .20; we can query the
+   UDR; we currently CANNOT get shell/logs on the affected endpoints). For each plausible outcome, what it tells you.
+
+---
+## YOUR ROUND-1 ANSWER (for reference)
+# GuruRMM — agent command delivery failure: diagnostic dossier (neutral)
+
+The evidence suggests a highly selective delivery failure where **outbound (Server -> Agent) WebSocket Text frames** are being dropped, mangled, or ignored, while **WebSocket Control frames (Ping)** and **inbound (Agent -> Server) traffic** remain functional.
+
+### 1. Candidate Root-Cause Hypotheses
+
+| Hypothesis | Reasoning / Evidence FOR | Reasoning / Evidence AGAINST |
+| :--- | :--- | :--- |
+| **H1: MTU / Fragmentation (Large Frame Blackhole)** | Common in site-to-site tunnels (UniFi UDR). Large commands (4KB) + overhead might exceed path MTU, causing fragmentation. If the path drops fragments, the Text frame never arrives. | **Refuted by T2/T3/T9:** Tiny `hostname` commands (~80B) also failed. MTU issues usually spare small packets. |
+| **H2: Application-Layer Deadlock (Agent Rx Loop)** | If the Agent's `handle_server_message` or the internal `tx` (to the executor) blocks or panics without crashing the read loop, the agent stops processing Text frames but continues to receive Pings (keeping the 90s deadline alive). | **Weakened by T7/T8:** A server-forced eviction and fresh reconnect (fresh Agent state) did not restore functionality. |
+| **H3: Intermediate Proxy / Firewall (Text-Frame Filtering)** | Cloudflare or a local "Security Gateway" (UniFi) may be inspecting WebSocket payloads. If the Command JSON pattern matches a signature or if the `Upgrade` handshake is being "half-policed," Text frames might be dropped. | The fleet baseline (40 agents working) suggests it's not a global Cloudflare issue, but could be specific to "enhanced" security profiles at certain sites. |
+| **H4: Protocol Desync (ConfigUpdate Serialization)** | On connect, the Server sends a `ConfigUpdate` (multi-KB). If this frame is malformed or if `tokio-tungstenite` on the server-side has a buffer issue with specific frame sequences, the Agent's parser might be stuck/errored for that specific stream. | **Supported by T8:** T8 confirms that even after a reconnect, delivery fails. This points to a failure during or immediately after the handshake/ConfigUpdate sequence. |
+| **H5: Nginx/Proxy Buffer "Clogging"** | `proxy_read_timeout` is long, but `proxy_buffers` or internal memory limits on the origin/nginx could be hit for specific connections if the socket becomes "write-blocked" from the origin to Nginx. | **Against:** Nginx logs would typically show 499s or buffer overflows; `send_to` on the server returns `success`, meaning it's reaching the OS/loopback buffer. |
+
+---
+
+### 2. The Verdict on the Evidence
+
+*   **Inference Check:** The claim that "the agent was receiving SOME inbound frames" (T4) is an **unverified inference**. While the 90s timeout didn't trigger, we cannot distinguish between "the agent received a Ping" and "the agent's internal timer is somehow stalled/incorrect."
+*   **Contradiction:** T8 is the most troubling. If a fresh connection (after eviction) doesn't fix it, the issue is either **environmental (Network/Path)** or **intrinsic to the Agent's 0.6.63 startup logic** (e.g., it crashes its own Text handler immediately upon receiving the first `ConfigUpdate` or `Command`).
+
+---
+
+### 3. The SINGLE Most Diagnostic Test
+
+**Action:** Dispatch a **WebSocket Ping from the Server** to a stuck agent and **explicitly log the receipt of the corresponding Pong**.
+
+**Reasoning:**
+*   **If Pong is received:** We confirm the path is fully transparent to WebSocket *Control* frames. This isolates the failure to **Text-frame serialization, size, or content inspection** (H3/H4).
+*   **If Pong is NOT received:** The agent is "ghosting" the server entirely. This means the 90s timeout in the agent (T4) is the actual failure point — the agent *should* have reconnected but didn't. This points to an **Agent-side Rx loop deadlock** (H2).
+
+**Outcome-based path:**
+*   **Pong Received:** Inspect Nginx/Cloudflare for "Web Application Firewall" (WAF) drops or "WebSocket inspection" settings.
+*   **Pong NOT Received:** The agent is dead-locked. Investigate the `tokio` task management in the Agent's 0.6.63 read loop.

tmp/r2-gemini.txt

@@ -0,0 +1,50 @@
+# GuruRMM command-delivery diagnosis — ROUND 2 (topology-corrected)
+
+The revelation that `rmm-api` is **grey-clouded** (direct to IP) and passes through **two nginx layers** (NPM + Origin) significantly shifts the focus. We can now discard Cloudflare-side buffering or WAF interference as the primary cause.
+
+### 1. REVISED ranked hypotheses
+
+| Hypothesis | Status | Reasoning |
+| :--- | :--- | :--- |
+| **H1: Nginx-to-Nginx Buffer Clog (Backpressure)** | **UP** [NEW TOP] | With two nginx layers in series, the "write-side" of the chain is more complex. If the Origin nginx has `proxy_buffering on` (default) and a small buffer, but NPM is slow to consume from the LAN, or the Agent is slow to consume from the WAN, the Origin nginx can "clog." The Rust server sees a successful write to the local socket, but the frame sits in an nginx buffer that never flushes to the next hop. |
+| **H2: Application-Layer Deadlock (Agent Rx Loop)** | **STABLE** | Still highly plausible. If the Agent's message-processing task (Tokio) hangs or panics during the handling of a specific message type (e.g., `ConfigUpdate` or a specific `Command` structure), the WebSocket read-loop might stay alive (responding to Pings at the library level) while the application logic stops processing `Text` frames. |
+| **H3: Path-Specific TCP/TLS Policing (UniFi/UDR)** | **UP** | Since there is no CDN normalizing the traffic, the endpoint gateway (UniFi UDR) is the first "intelligent" hop. If "Deep Packet Inspection" (DPI) or "IDS/IPS" is enabled on the UDR, it may be flagging the specific entropy/pattern of your encrypted WebSocket Text frames while ignoring the predictable patterns of Ping/Pong control frames. |
+| **H4: Protocol Desync (Nginx `proxy_http_version`)** | **DOWN** | The Origin config explicitly sets `proxy_http_version 1.1`, which is correct for WebSockets. Unless NPM is downgrading the connection to 1.0 (highly unlikely for a WS-enabled proxy), this is less likely than a buffering issue. |
+| **H5: MTU / Fragmentation** | **DROPPED** | As established in Round 1, the failure of ~80B commands (`hostname`) effectively refutes MTU as the primary culprit. |
+
+---
+
+### 2. The Verdict on the Architecture
+
+The **double-nginx hop** is a major red flag for WebSocket stability. WebSockets are long-lived and stateful; each proxy layer adds a potential point of "silent failure" where the TCP connection remains `ESTABLISHED` but the application-level framing is stuck.
+
+**Critical Observation:** The fact that one command succeeded *immediately* after a reconnect, followed by total silence, strongly suggests a **buffer saturation** or **one-time logic crash**. If it were a protocol mismatch, the first command would likely have failed too.
+
+---
+
+### 3. The SINGLE Most Diagnostic Test
+
+**Action:** Run a dual-ended packet capture on the **Origin Host (.30)** simultaneously on the **Incoming LAN interface** (traffic from NPM .20) and the **Loopback interface** (traffic to Rust server :3001).
+
+```bash
+# Terminal 1: Capture from NPM to Origin (Inbound to .30)
+tcpdump -i eth0 -n -A src 172.16.3.20 and port 80
+
+# Terminal 2: Capture from Origin to Rust (Inbound to :3001)
+tcpdump -i lo -n -A port 3001
+```
+
+**Why this is the "Golden" test:**
+By comparing the two captures during a command dispatch, you identify exactly where the "Text" frame disappears in the chain:
+
+*   **Outcome A: Command appears in BOTH captures.**
+    *   **Meaning:** The entire server-side infrastructure is working. The command was handed to the OS to be sent over the WAN.
+    *   **Verdict:** The issue is **H2 (Agent Deadlock)** or **H3 (ISP/Gateway interference)**.
+*   **Outcome B: Command appears in Loopback (:3001) but NOT on eth0.**
+    *   **Meaning:** The Rust server sent it, the Origin nginx received it, but the Origin nginx failed to forward it to NPM.
+    *   **Verdict:** **H1 (Nginx Buffering)**. You likely need `proxy_buffering off;` on the Origin nginx.
+*   **Outcome C: Command appears in NEITHER capture.**
+    *   **Meaning:** The Rust server's `send` call is lying or the frame is stuck in the Rust server's internal `tokio-tungstenite` sink.
+    *   **Verdict:** Internal Rust server-side concurrency/async issue.
+
+**Recommendation:** While running the capture, dispatch a **Ping** as well. If the Ping passes through both captures but the Command (Text) does not, you have confirmed **Outcome B** and isolated it to an Nginx `Text` frame handling/buffering issue.

tmp/r2-grok-prompt.md

@@ -0,0 +1,222 @@
+# GuruRMM command-delivery diagnosis — ROUND 2 (important topology correction)
+
+You analyzed this in round 1 (your prior answer is included at the bottom). A FACTUAL CORRECTION
+to the architecture has come to light that invalidates part of the round-1 premise. Please
+re-evaluate from scratch where needed. Be willing to discard your prior top hypothesis.
+
+## CRITICAL CORRECTION to the ingress topology
+
+In round 1 we told you: "agent -> Cloudflare -> nginx -> Rust server". **That was wrong for the
+AGENT path.** Verified facts now:
+
+- The agent's hard-coded WebSocket URL is **`wss://rmm-api.azcomputerguru.com/ws`** (from agent source:
+  `DEFAULT_SERVER_URL`, and the agent config default). The installer and enrollment also use
+  `rmm-api.azcomputerguru.com`.
+- DNS / Cloudflare proxy status for the two hostnames (both A-record to the same public IP 72.194.62.10):
+  - `rmm.azcomputerguru.com`     -> **proxied = true**  (orange cloud; goes THROUGH Cloudflare)  — this is the human DASHBOARD.
+  - `rmm-api.azcomputerguru.com` -> **proxied = false** (grey cloud; **DNS-only, BYPASSES Cloudflare**) — this is what the AGENTS use.
+- Therefore **Cloudflare is NOT in the agent's path at all.** All round-1 hypotheses about Cloudflare
+  WAF / CF WebSocket buffering / CF frame handling are moot for agent command delivery. (Cloudflare only
+  fronts the dashboard.)
+
+## The ACTUAL agent path (verified)
+
+```
+agent (endpoint, e.g. PST-SERVER behind a UniFi UDR/"Cloudflare-Ultra"-class gateway)
+  -> endpoint LAN/NAT
+  -> public internet
+  -> 72.194.62.10  (public IP; this is the NPM box)
+  -> NPM = "Nginx Proxy Manager" on host 172.16.3.20  (terminates TLS; one nginx layer)
+       NPM proxy-host settings for rmm-api: forward http://172.16.3.30:80, websockets=ON,
+       http2_support=ON, block_exploits=ON (NPM "Block Common Exploits"), caching=OFF, advanced_config=EMPTY
+  -> http://172.16.3.30:80   (the ORIGIN nginx; a SECOND nginx layer)  -- PLAINTEXT HTTP over the LAN here
+  -> proxy_pass http://127.0.0.1:3001   (the Rust server)
+```
+
+So there are **TWO nginx proxy layers** in series (NPM on .20, then origin nginx on .30), no CDN.
+
+Origin nginx `/ws` block (verbatim):
+```
+location /ws {
+    proxy_pass http://127.0.0.1:3001;
+    proxy_http_version 1.1;
+    proxy_set_header Upgrade $http_upgrade;
+    proxy_set_header Connection "upgrade";
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_read_timeout 86400;
+}
+```
+(No explicit `proxy_buffering off;`, no `proxy_send_timeout`. NPM's generated config for the proxy host is
+the standard NPM template with websockets enabled + block_exploits; we have not yet dumped its exact directives.)
+
+## What this changes / things to reconsider
+
+- The "asymmetry" symptom is unchanged and still the core puzzle: for affected agents the server keeps
+  receiving inbound (heartbeats, last_seen fresh), the agent does NOT trip its 90s no-inbound reconnect
+  (so it appears to be receiving at least the server's 30s WS Ping frames), yet server->agent COMMAND (Text)
+  frames are never acked/executed — both ~4 KB commands AND ~80 B `hostname` commands. One command DID
+  succeed earlier (right after the agent updated/reconnected). A server-forced eviction + fresh reconnect did
+  NOT restore command delivery (a tiny command on the fresh connection still never acked) — but note we have
+  NO confirmation the agent received anything server->agent on that fresh connection either.
+- Affected: PST-SERVER + PST-SERVER2 (same site/UDR), and GTS-PEDRO-H (different site). ~40 other agents
+  deliver commands fine through this SAME NPM+nginx path.
+- Because rmm-api is grey-cloud, the agent's TLS connection goes straight from the endpoint's gateway to the
+  public IP — there is no CDN normalizing MTU/keepalive/buffering on the path; the endpoint's own gateway
+  (e.g. PST's UDR) is directly in the path.
+
+## A note on one earlier "test"
+
+We tried to `tcpdump` the origin loopback (nginx->Rust on 127.0.0.1:3001) during a marked command dispatch and
+saw 0 packets — but that capture almost certainly FAILED (the backgrounded tcpdump was killed when the SSH
+session closed), so treat it as NO DATA, not as evidence that the loopback is silent.
+
+## Please now provide (round 2)
+
+1. Your REVISED ranked hypotheses given Cloudflare is out of the agent path and there are two nginx layers +
+   the endpoint gateway directly in the path. Explicitly say which round-1 hypotheses you drop or demote.
+2. The single most diagnostic next test (we can run packet captures on the .30 host on the plaintext .20->.30:80
+   leg and on the 127.0.0.1:3001 loopback; we can read NPM's generated nginx config on .20; we can query the
+   UDR; we currently CANNOT get shell/logs on the affected endpoints). For each plausible outcome, what it tells you.
+
+---
+## YOUR ROUND-1 ANSWER (for reference)
+# Diagnostic analysis: GuruRMM command delivery failure
+
+## What the evidence actually proves
+
+**Established facts:**
+- Agent → server path is healthy for affected agents: `last_seen` stays current, Heartbeats arrive, re-offer logic fires on Heartbeat.
+- Server believes it is dispatching: `send_to()` returns true, reaper logs re-delivery, no enqueue failures.
+- Affected agents never send `CommandAck` and never execute commands (large and tiny).
+- Most of the fleet (~40 agents, 397 commands in 6h) is unaffected on the same server build.
+- PST-SERVER had one successful acked command at 22:23:40 on 0.6.63; failure begins after the 00:22:03 reconnect.
+- Eviction → reconnect works mechanically on the server side, but commands still fail on the fresh connection (T8).
+
+**Unverified inferences (treat as hypotheses, not facts):**
+- “Agent is receiving server Pings” — inferred only from *no* 90s agent-side reconnect (T4). No Pong logging on the server; server code ignores inbound `Pong`.
+- “`send_to` success means the command was sent” — it only means the in-memory channel accepted the message.
+- “ConfigUpdate was delivered on reconnect” — server logged `Sent ConfigUpdate`; no agent-side confirmation.
+- PST-SERVER, PST-SERVER2, and GTS-PEDRO-H share one root cause — GTS failed with `attempts=1` at a 60s timeout, not the 10-attempt undeliverable pattern; it had been idle since 06-06. That may be a second failure mode.
+
+**The core puzzle:** This is not a dead connection in the usual sense. It looks like a **live, asymmetric channel**: inbound app traffic works; outbound *command delivery* does not — yet the connection is stable enough that the agent does not trigger its 90s “no inbound activity” reconnect.
+
+---
+
+## Ranked root-cause hypotheses
+
+### 1. Server outbound Text writes succeed at enqueue but fail (or stall) at the socket — while Ping writes still work via `select!` interleaving
+**For:**
+- `send_to()` only checks `tx.send()` on an (likely unbounded) channel, not socket delivery.
+- Send task uses one loop for Text and Ping; a **stuck `sender.send(Text).await`** does not necessarily kill the task immediately; Ping ticks can still fire when not blocked inside a Text send.
+- Would explain: heartbeats in, no CommandAck, no execution, reaper climbing attempts, “online” agent.
+- Fresh reconnect (T8) could still fail if the underlying TCP/WebSocket path is half-open for application Text but still passes occasional Ping frames.
+
+**Against:**
+- If the send task were **permanently** blocked on Text, Pings would stop and the agent should reconnect every ~90s. T4 shows a 30+ minute stable connection — so either Pings are getting through, or the 90s inference is wrong.
+- After eviction, a brand-new TCP session should reset middlebox state; T8 still fails, which weakens “stuck write on old socket” unless the pathology is immediate on every new session at that site.
+
+**Verdict:** Strong, but needs socket-level write confirmation. The contradiction around 90s stability is the main gap.
+
+---
+
+### 2. Agent receives inbound frames (at least Pings) but server → agent **Text** frames are dropped or never parsed — agent-side or on-path selective loss
+**For:**
+- Best fit for “connection alive, heartbeats out, commands never acked/executed.”
+- Agent ACKs on receipt *before* execution; no ACK + no execution ⇒ Command handler likely never ran on a parsed `Command`.
+- Tiny `hostname` (~80 B) fails (T2, T3, T8) — rules out payload size as the primary filter.
+- PST-SERVER regressed exactly at reconnect — first thing on a new session is `AuthAck`, `ConfigUpdate`, then command re-offers. We do not know whether *any* post-reconnect Text reached the agent.
+
+**Against:**
+- WebSocket middleboxes rarely drop only `Text` opcode frames while passing `Ping`. Possible but unusual unless something is inspecting JSON content.
+- Does not by itself explain GTS-PEDRO-H at a different site unless this is a broader server bug in Text serialization for certain agents.
+
+**Verdict:** Very plausible. “Ping works, Text doesn’t” is an inference, not a measurement — but the symptom shape matches.
+
+---
+
+### 3. Agent-side bug or state corruption in 0.6.63 (Command path broken; Heartbeat/metrics path fine)
+**For:**
+- PST-SERVER2 has **never** acked a command on 0.6.63 — suggests a persistent local pathology, not a transient network blip.
+- PST-SERVER worked once, then failed after reconnect — consistent with handshake/`ConfigUpdate` leaving the command handler or ACK channel in a bad state.
+- Agent architecture likely separates heartbeat/metrics tasks from the read/command path; one can work while the other is broken.
+- Explains T8: new server connection does not help if the agent process state is wrong.
+
+**Against:**
+- Hard to explain why 40 other agents on 0.6.63 are fine unless trigger is site-specific policy/config in `ConfigUpdate`.
+- Without agent logs, this is hard to distinguish from hypothesis #2.
+
+**Verdict:** High plausibility, especially for PST-SERVER2. May combine with #2 (bad config triggers bug).
+
+---
+
+### 4. Site/gateway pathology (UniFi UDR “Cloudflare-Ultra” class) affecting server → client WebSocket application data
+**For:**
+- PST-SERVER and PST-SERVER2 share site and gateway; both broken, overlapping timeframe.
+- UniFi + aggressive DNS/CF integration could plausibly affect long-lived WSS in non-obvious ways.
+
+**Against:**
+- GTS-PEDRO-H is a different site and also fails — unless that’s unrelated (see #6).
+- Fleet-wide Cloudflare + nginx path works for 40 agents; edge config would need to be connection- or path-specific.
+- Tiny frames also fail — less consistent with typical MTU/DPI size limits.
+
+**Verdict:** Plausible for PST pair alone; weak as a single explanation for all three agents.
+
+---
+
+### 5. Stale/wrong connection routing in server in-memory agent map (commands enqueued to a dead sender; heartbeats handled elsewhere)
+**For:**
+- Classic split-brain pattern for connection registries.
+- `send_to` returning true on a queued-but-never-drained channel matches symptoms.
+
+**Against:**
+- Eviction explicitly closed the connection and re-ran handshake (T7); map should have been replaced.
+- Single nginx → single Rust server process; no load-balancer ambiguity described.
+- If receive task is dead, `last_seen` would not update — it does.
+
+**Verdict:** Lower plausibility after T7/T8, unless there is a subtle duplicate-registration bug not ruled out by eviction.
+
+---
+
+### 6. Multiple independent failure modes grouped by coincidence
+**For:**
+- GTS-PEDRO-H: 6-day command gap, `attempts=1`, 60s timeout — different from PST’s climb to 10 “undeliverable.”
+- Pre-06-12 successes with `acked=false` are expected (pre-CommandAck).
+- One test on GTS (T9) is thin evidence to merge it with PST.
+
+**Against:**
+- All three are 0.6.63 and fail the same operator test (`hostname`, no ack).
+
+**Verdict:** Do not assume one root cause for all three without more data. GTS may be a red herring or a second bug.
+
+---
+
+### 7. Command JSON / server message construction bug affecting only certain agents
+**For:**
+- Could cause agent to silently drop unparsable `Text` while still handling other message types — if those other types were received, which we have not confirmed post-reconnect.
+
+**Against:**
+- Same server code path completes 397 commands for other agents.
+- Would need agent-specific command content or policy embedding in the `Command` frame.
+
+**Verdict:** Low unless `ConfigUpdate` policy differs per agent and corrupts something downstream.
+
+---
+
+## Single most diagnostic next step
+
+**Instrument the server per-connection send task and receive loop for one affected agent (PST-SERVER): log (with monotonic timestamps and `connection_id`) every completed `sender.send()` by frame type (`Text` vs `Ping`), Text byte length and `command_id`, and every inbound `Pong` correlated to the preceding `Ping`; then dispatch one tiny `hostname` command.**
+
+No agent access required. Run for 2–3 minutes on the stuck agent.
+
+| Outcome | Interpretation |
+|--------|----------------|
+| **Ping writes succeed + inbound Pongs observed; Text write for `hostname` logs success; still no `CommandAck`** | Server → socket → nginx path delivers control frames; **agent is not processing server Text** (agent bug, or on-path Text opcode loss after origin). Strongest signal for #2 / #3. |
+| **Ping writes succeed + Pongs observed; Text write errors or never completes** | **Server-side or loopback socket pathology** on application Text (#1). |
+| **Ping writes logged but no inbound Pongs** | The “agent receives Pings” inference was wrong; connection is stranger than modeled — revisit half-open TCP, nginx connection mapping, or agent timeout logic (#1 / #5). |
+| **Neither Ping writes nor Pongs, yet agent stays “online” and heartbeats continue** | **Split connection or duplicate session**: inbound and outbound are not the same WebSocket the send task thinks (#5). Escalate to connection registry + nginx upstream connection audit. |
+| **Text write succeeds on fresh post-eviction `connection_id`, same as above** | Rules out “stale socket only”; points to **persistent agent-side or site-path Text loss** (#2/#3/#4). |
+
+That one test separates “server thinks it sent” from “bytes left the process,” validates or falsifies the Ping inference, and narrows the fault domain to **origin socket / agent / middlebox** without needing endpoint access first.
+
+**Secondary high-value data (if the above is already partially instrumented):** nginx `tcpdump` or OpenResty debug on loopback `:3001` during a `hostname` dispatch — confirms whether Text frames leave the Rust process into nginx. That does not prove WAN delivery, but it collapses #1 vs #2/#3 on the origin host in one capture.

tmp/r2-grok.txt

@@ -0,0 +1,97 @@
+**Round 2 revised analysis (Cloudflare removed from agent path; two nginx layers + endpoint gateway now primary surface)**
+
+### Key factual corrections applied
+- Agent path is DNS-only (grey cloud) to `rmm-api.azcomputerguru.com`. Cloudflare is completely out of the agent data plane. All prior hypotheses that invoked CF WAF, CF WebSocket buffering, CF frame handling, or CDN normalization of MTU/keepalives are dropped.
+- Explicit two-nginx chain for agents: NPM (172.16.3.20, TLS termination, standard WS-enabled proxy host) → origin nginx (172.16.3.30:80, plaintext) → `proxy_pass http://127.0.0.1:3001`.
+- Origin `/ws` location supplies the basic upgrade headers and a long `proxy_read_timeout`, but lacks `proxy_buffering off;`, any `proxy_send_timeout`, `proxy_cache off`, or other streaming/WS hardening. NPM settings include `websockets=ON`, `http2_support=ON`, `block_exploits=ON`, caching=OFF, empty advanced config.
+- The endpoint gateway (e.g. PST UniFi UDR) sits directly on the WAN leg with no CDN in front.
+- Symptom core is unchanged and still asymmetric: agent→server (heartbeats, last_seen, re-offer triggering) works; server believes it dispatched (send_to true); tiny (~80 B) and larger commands produce no CommandAck and no execution; the agent does not trip its 90 s no-inbound reconnect; one early success occurred right after update/reconnect; forced eviction + fresh reconnect reproduced the failure for a subsequent tiny command.
+- ~40 other agents succeed through the identical NPM+origin path; the three affected (two sharing a UDR, one at a different site) are on 0.6.63.
+
+### Revised ranked hypotheses
+Ranked by current fit to the corrected topology and the persistent asymmetry. I explicitly note changes from round 1.
+
+**1. Origin nginx (and/or NPM) proxy configuration is insufficient for reliable server→client WebSocket data-frame delivery after upgrade.**  
+Missing `proxy_buffering off;` (nginx default is on) plus absent send timeout, combined with two proxy layers in series and NPM's block_exploits + http2_support, can cause server-originated Text frames to be buffered, not flushed promptly, or mishandled while small control frames (Pings) and the reverse direction (client heartbeats) continue. The upgrade handshake succeeds and the connection stays "live" from the origin's viewpoint, but application Text is not reliably forwarded to NPM's client-facing side.  
+**For:** Exactly matches asymmetry + "Pings appear to arrive (no 90 s reconnect) but commands never do"; both tiny and large Text fail; fresh TCP session after eviction still uses the same proxy config so reproduces; early post-reconnect success is consistent with transient "hot" proxy state before buffering or idle behavior kicks in. The provided origin config is the minimal common template, not the hardened WS template.  
+**Against:** 40 agents work on the same path, so any defect must be intermittent, connection-age-dependent, timing-dependent, or only triggered for certain agents' command dispatch patterns.  
+**Verdict:** Now the single strongest hypothesis. Round-1 #1 (server socket stall) is reframed here because the two visible nginx layers are the new, obvious place where Text vs. control or directional asymmetry can be introduced without killing the whole connection.
+
+**2. Agent 0.6.63 command receive / Ack path is broken or left in a bad state while the heartbeat write path and control-frame handling remain functional.**  
+The WS reader task processes Pings (or at least whatever keeps the 90 s timer happy) and can still emit heartbeats, but never delivers a parsed `Command` Text frame to the handler (or the Ack path is one-way dead). Could be triggered by ConfigUpdate content, a reconnect sequence, or per-agent state.  
+**For:** Explains why only a subset of 0.6.63 agents are affected, why eviction + new server-side connection does not help (agent process state persists), why both sizes fail, and the "worked once then stopped" timeline on PST-SERVER. Heartbeats out are a separate task from inbound command dispatch.  
+**Against:** Same build works for the large majority through the identical infrastructure; would require something agent-specific (config, enrollment data, or a race that only some hit).  
+**Verdict:** Still very plausible and hard to rule out without agent visibility. Promoted slightly relative to round 1 because the network path is now better understood and the symptom remains after clean reconnects.
+
+**3. Endpoint gateway (UDR at PST site; whatever is at GTS-PEDRO-H) selectively interferes with server→client WebSocket Text frames on the direct WAN leg.**  
+Stateful inspection, DPI/IPS, TCP flow tracking, or long-lived-connection handling in the gateway drops, delays, or mangles downstream application data frames (Text opcode) while passing control frames (Ping) and upstream traffic. No Cloudflare means the gateway's behavior is unmediated.  
+**For:** PST-SERVER and PST-SERVER2 share the same UDR and site; the direct internet path puts the gateway squarely in the frame path. Early success followed by persistent failure is consistent with a flow entry being installed with a bad timeout or inspection state.  
+**Against:** Tiny 80 B command also fails (argues against pure MTU/fragmentation); GTS-PEDRO-H is a different site (unless it has similar gateway hardware/policy); 40 other agents with their own gateways work.  
+**Verdict:** Plausible primary cause for the PST pair; weaker as a unified explanation for all three unless the gateways are alike. Round-1 site/gateway hypothesis is retained but narrowed (no Cloudflare involvement).
+
+**4. Server-side outbound send task or WS library usage silently fails (or starves) Text frames for commands while continuing to emit Pings and accept inbound frames.**  
+`send_to()` only enqueues to an in-memory channel; the actual tungstenite/etc. write for Text either never completes, is not flushed, targets a stale sender, or is deprioritized behind Pings in the select!/timer loop.  
+**For:** Classic "enqueue reports success, bytes never leave." Would explain reaper behavior and "online but no delivery."  
+**Against:** Eviction + fresh reconnect (T8) should have produced a brand-new sender and socket; same server binary successfully delivers hundreds of commands to other agents on the same path. Round-1 version of this hypothesis is demoted because the new-connect failure and removal of Cloudflare weaken the "old socket only" story.  
+**Verdict:** Lower than in round 1 but not eliminated; a subtle difference between how Command Text vs. Ping frames are written/serialized/flushed remains possible.
+
+**5. NPM-generated config or runtime behavior for the rmm-api proxy host does not correctly forward server→client WS data frames post-upgrade.**  
+Even with `websockets=ON`, the combination of http2_support, block_exploits, or the standard NPM template may only partially upgrade the backend connection or may apply buffering/inspection to the NPM↔origin leg that affects one direction or one opcode.  
+**For:** NPM is the first hop after the internet and the TLS terminator; its config is a black box until dumped. block_exploits sounds like it could contain request/response or header inspection that survives the upgrade in unexpected ways.  
+**Against:** If the WS proxying were fundamentally broken, far more (or all) agents would be affected.  
+**Verdict:** Worth immediate checking via config dump; likely to combine with #1 rather than stand alone.
+
+**6. Stale or duplicate connection registration in the Rust in-memory agent map (commands routed to a sender no longer attached to the live WS sink).**  
+**For:** Perfectly explains "send_to true, nothing arrives, receive side still updates last_seen."  
+**Against:** Explicit eviction + re-handshake (T7/T8) should have replaced the entry; single-process server, single nginx path.  
+**Verdict:** Strongly demoted from round 1. The fresh-connect failure makes registry split-brain much less likely unless there is a subtle map-key or cleanup bug not exercised by normal eviction.
+
+**7. Multiple independent root causes (GTS-PEDRO-H is a separate bug from the PST pair).**  
+GTS showed `attempts=1` + 60 s timeout after a long idle period; PST shows climbing to 10 "undeliverable." Different sites.  
+**Verdict:** Retain as a live possibility for GTS until the captures or more data merge or separate them. The shared "tiny hostname command, no ack, 0.6.63" observation argues against treating them as wholly unrelated.
+
+Dropped from round 1 (explicitly):  
+- All Cloudflare hypotheses (WAF, CF buffering, CF WebSocket frame handling, CDN MTU/keepalive normalization). These no longer apply to the agent path.  
+- Pure "long-lived socket half-open only on old connections" framing is demoted (T8 fresh reconnect reproduced the symptom).  
+- Any assumption that "Cloudflare is in the path and could be normalizing or interfering" is removed.
+
+The core puzzle is now cleanly: bytes for server→agent Text either never leave the Rust process, are not forwarded by one or both nginx layers, are lost or mangled by the endpoint gateway, or are received by the agent but not turned into CommandAck + execution.
+
+### Single most diagnostic next test
+**Dump NPM's exact generated nginx config for the rmm-api proxy host + coordinated packet capture on the .30 host (both legs) during a marked tiny command dispatch to a known-stuck agent (PST-SERVER).**
+
+**Execution:**
+- On the .20 (NPM) host, locate and export the full server block / location for rmm-api.azcomputerguru.com (typically under `/etc/nginx/proxy_host/...` or the live conf). Capture at minimum: listen directives, the location / (or /ws) block, all proxy_* settings, any upgrade/connection headers, buffering/cache/timeout settings, http2 directives, and any reference to block_exploits or advanced config.
+- On the .30 host, with PST-SERVER connected and in the failing state:
+  - Start two background captures (pcap preferred for full frames; add `-vv -X` or similar text output if you need immediate readability):
+    - Loopback: traffic involving port 3001 (Rust ↔ origin nginx). `tcpdump -i lo -s 0 -w /tmp/loopback-3001.pcap 'port 3001'`
+    - LAN leg: traffic between .30 and .20 on port 80. Identify the interface with 172.16.3.30 (or the subnet), then `tcpdump -i <if> -s 0 -w /tmp/lan-80.pcap 'host 172.16.3.20 and port 80'`
+  - Optionally raise origin nginx log level (error/access) for the duration if it is not already informative.
+  - From the server side (admin/test path or direct), dispatch one uniquely identifiable tiny command (hostname with a nonce/timestamp in command_id or payload if possible) targeted only at PST-SERVER.
+  - Wait 60–90 s (covers reaper cycles).
+  - Stop captures, collect the pcaps + origin nginx logs for the relevant minute(s) + server-side logs for that connection_id (every outbound Text/Ping with size/command_id, every inbound frame including heartbeats and any Pongs).
+- (If the UDR web UI or API exposes it) query active sessions, flow table, or recent traffic logs for the public server IP or the agent's known public IP around the dispatch timestamp.
+
+**Analysis focus (use tshark/wireshark or text dump):**
+- Does the loopback capture show the WS Text frame (opcode 0x81) containing the command leaving the Rust side toward 3001?
+- Does the LAN capture show the corresponding data shortly afterward on the established TCP stream from .30 → .20 (the same stream that carried the original WS upgrade)?
+- Are inbound heartbeats visible in both captures (client → .20 → .30 → Rust)?
+- Are server Ping control frames (opcode 0x89) visible leaving Rust and/or crossing to .20?
+- Any RST/FIN, zero window, or errors around the time?
+- Cross-reference exact timestamps with server "sent" logs.
+
+**Outcome table and interpretation**
+
+| Outcome (what the captures + config show) | Interpretation |
+|-------------------------------------------|----------------|
+| No Text frame for the dispatched command appears on loopback from Rust (server log may still claim send_to success); heartbeats arrive on loopback. | Failure is inside the Rust process (send task never actually wrote the frame to the socket for this connection, wrong sender, flush not happening, or enqueue-only success). Points to revised #4 (server outbound Text path). Rules out nginx/gateway/agent for this dispatch. |
+| Text frame visible leaving Rust on loopback to origin nginx, but no corresponding WS data seen in LAN capture from .30 to .20 on the proxy conn (inbound heartbeats visible both places). | Origin nginx accepted the frame on 3001 but did not forward it (or buffered it indefinitely) to its socket toward NPM. Strong evidence for inadequate proxy config at origin (#1) — especially the missing `proxy_buffering off;`, lack of send timeout, or interaction with the two-layer setup. |
+| Text frame visible on both loopback and LAN (left .30 toward NPM). Agent still never acks/executes. | Bytes traversed both nginx layers. Problem is downstream of .30: NPM did not relay over the TLS client socket (#5), or the frame was emitted by NPM but dropped/mangled by the endpoint gateway or on-path devices (#3), or the agent received it at the WS layer but did not parse/ack/execute (#2). Next actions become: capture at .20 if possible, UDR flow inspection, or agent-side instrumentation in a later build. |
+| Pings (control) from server visible on loopback and/or LAN; client heartbeats flow normally; the specific command Text is absent or truncated at one of the capture points. | Selective handling of Text (data) opcode vs. control frames or vs. client→server direction. Reinforces proxy-layer (#1) or gateway (#3) as the filter. |
+| Full bidirectional frame flow visible in captures (heartbeats, Pings if emitted, the command Text leaves Rust and crosses to .20), yet no CommandAck ever arrives at the server. | The infrastructure up to NPM forwarded the frame. Either NPM→WAN→gateway→agent path lost it, the agent ignored it, or (less likely) the capture timing missed a very delayed delivery. This would elevate #2 or #3 and de-emphasize the nginx config. |
+| Captures + server logs show normal inbound activity and that the server did emit Pings, but no Pongs are ever observed for those Pings, yet the agent remains "online" with fresh last_seen and does not 90 s reconnect. | The round-1 inference that "the agent must be receiving server Pings" is weaker than assumed. The agent's no-inbound timer may be driven only by app-level heartbeats, by any outbound activity, or may simply be longer/more forgiving. Does not prove the connection is fully healthy for server→agent data. |
+| No evidence the command was even attempted in server logs for the live connection_id (despite operator dispatch), while heartbeats continue to update last_seen on that same connection. | Registry / send_to targeting issue (#6). The eviction did not fully replace the live sender for commands. |
+
+This single coordinated step (config dump + dual-leg capture on .30 during one marked dispatch) directly answers the three critical questions without any endpoint access: (1) did bytes leave the Rust process?, (2) did the origin nginx forward them across the plaintext leg to NPM?, (3) was the proxy config obviously deficient? It will confirm or largely eliminate the two-nginx configuration surface (#1/#5) and sharply narrow the remaining possibilities to server, gateway, or agent. If the frames cleanly exit .30, the next practical steps become NPM-side visibility or a controlled bypass of one nginx layer for a test agent.
+
+The asymmetry after a clean reconnect is the fact that continues to make pure "old socket" or "stale registry" stories less attractive; the data now points most strongly at either the visible proxy layers or an agent-side receive/handler defect that survives reconnection. The captures will decide.

tmp/r3-gemini.txt

@@ -0,0 +1,35 @@
+[INFO] Independent Reviewer Verdict: The "NAT-straight-to-.30" proposal is a high-effort diagnostic that provides an "unclean" bisection. It should be deferred in favor of a 5-minute configuration change and a packet capture.
+
+### 1. Diagnostic Bisection Analysis
+- **If it works (commands resume):** It proves the failure occurs at the NPM layer (.20) or in the NPM-to-Origin network leg. **Critique:** It does *not* distinguish between an Nginx configuration error (H1) and an NPM-specific resource exhaustion or OS-level socket leak.
+- **If it fails (still no commands):** It exonerates NPM. The issue is localized to the origin Nginx, the Rust server, the Agent (H2), or the Gateway (H3).
+- **Confounds:** This is NOT a clean bisection. Moving TLS termination from NPM to Origin changes the TLS stack (OpenResty vs. Nginx), cipher suites, and TCP window behavior. If the Gateway (H3) is dropping packets based on specific TCP/TLS fingerprinting or MTU issues triggered by NPM's headers, the "fix" might be a side effect, not a localization.
+
+### 2. Permanent Fix: Pros and Cons
+- **Pros:**
+    - Reduces architectural complexity and "double-proxying" overhead.
+    - Eliminates one potential point of failure and logging fragmentation.
+- **Cons:**
+    - **Security Surface:** .30 is now directly exposed. NPM acts as a sacrificial buffer; losing it moves the "front line" to the application host.
+    - **Cert Management:** You must migrate and automate Certbot/ACME on .30. If this is manual, you risk expiry-related downtime.
+    - **Fragmentation:** If the Cloudflare-fronted dashboard stays on NPM, you now have two different TLS entry points and two different Nginx configs to maintain. 
+    - **Cloudflare Integration:** You lose the ability to use NPM’s GUI to quickly toggle ACLs or WAF settings for the agent path.
+
+### 3. Lower-Risk / Faster Diagnostic Signals
+The current plan skips the "Evidence" phase and jumps to "Structural Change."
+- **A. Config Hardening (The "Magic" Flag):** Add `proxy_buffering off;` to the `/ws` block on BOTH NPM and Origin. Nginx often attempts to buffer small upstream responses (Text frames) but lets through TCP Pings (Binary/Lower-level). This is the most likely culprit for H1.
+- **B. Targeted PCAP (The Smoking Gun):** Since `tcpdump` is installed, run it on both .20 and .30 simultaneously:
+  `tcpdump -i any port 3001 or port 80 -w capture.pcap`
+  Send a command. If the frame is seen entering .20 but not leaving it, NPM is the killer. If it leaves .30 but isn't seen at the agent's WAN, H3 is confirmed.
+- **C. Single-Agent Pivot:** Instead of re-NATing all 200 agents, NAT a non-standard port (e.g., 4443) straight to .30. Point *one* affected agent to `wss://rmm-api.azcomputerguru.com:4443/ws`. This isolates the variable for PST-SERVER without risking the fleet.
+
+### 4. Net Recommendation
+[WARN] Do NOT re-NAT the production fleet yet. The risk of TLS/cert misconfiguration and downtime is high for a diagnostic gain that can be achieved more cheaply.
+
+**Ordered Action List:**
+1. **Apply `proxy_buffering off;` and `proxy_read_timeout 86400;`** to the `/ws` location blocks on both Nginx layers. Restart Nginx. (Time: 5 mins).
+2. **Execute a dual-sided `tcpdump` capture.** Verify if the Text frame (search for command payload strings in hex) survives the jump from .30 to .20. (Time: 15 mins).
+3. **Internal Bypass Test:** From a machine on the same LAN as the agents (if possible), use `wscat` or a similar tool to connect directly to .30:80 to see if commands arrive when TLS and NPM are bypassed locally.
+4. **Pivot only if Step 2 shows NPM swallowing frames.** If you must pivot, use the "Parallel Port" method (Step 3C) before moving the entire fleet.
+
+**Confidence:** 90% that the issue is `proxy_buffering` or a TCP keep-alive mismatch between the two Nginx layers, rather than a fundamental flaw in NPM itself.

tmp/r3-grok.txt

@@ -0,0 +1,75 @@
+**1. Diagnostic bisection value**
+
+Success (affected agents start receiving commands after NPM is removed and traffic goes direct to .30's single nginx):
+
+- This would be strong evidence that the problem lives in the current dual-nginx path (H1), most likely in or between NPM and the origin nginx. The fact that heartbeats (agent→server) and apparently server WS Pings continue working while Text frames do not already points to a server→agent data path issue; removing one entire proxy layer and its configuration would cleanly implicate that layer or the interaction between the two.
+- It would make H2 (pure agent 0.6.63 defect) much less likely, because the same agent binaries would now be receiving frames they previously did not. A reconnect already failed to fix it, so a pure agent-side stuck state is less plausible if the path change alone restores delivery.
+- H3 (UDR/gateway interference) would be largely ruled out for the affected agents, because the path from the UDR outward (to the public IP) is unchanged; only the internal destination after the public IP changes.
+
+Failure (agents still do not receive commands):
+
+- This would rule out "NPM + two nginx layers" as the sole cause. It would shift weight to H2 or H3, or to something that survives the change: the Rust server itself (send path or per-connection state), the specific nginx config that gets deployed on .30, the TLS termination behavior on .30 vs NPM, or some other invariant (e.g., the public IP, the outer NAT device, or a property of the long-lived connections from those specific agents).
+- It would not cleanly distinguish H2 from H3.
+
+**Confounds and why it is not a perfectly clean bisection**:
+- You are not just "removing NPM"; you are also moving TLS termination from NPM's nginx to a new (or newly configured) nginx instance on .30. Any difference in nginx version, compiled modules, exact location block, `proxy_*` directives, or even OpenSSL behavior becomes a variable.
+- The internal hop (.20 ↔ .30) disappears. If there was any subtle interaction (extra buffering, header rewriting, connection pooling oddity, or MTU/fragmentation effect on the internal segment), it is removed at the same time.
+- If the public IP's 443 is DNATed wholesale to .30, both the grey-cloud API and the Cloudflare orange-cloud dashboard now land on .30's nginx. Any difference in how Cloudflare-injected headers, real IP handling, or HTTP/2 are processed on .30 vs the current NPM setup becomes entangled.
+- The affected population is a small subset of agents on the same path that already work for ~40 others. A fix that only appears after a full topology + TLS-terminator change does not tell you whether the root cause was a per-connection state bug in NPM that only some flows hit, or something narrower.
+- In short: success would be good evidence against the current dual-layer setup; failure would be informative but would leave the remaining hypotheses entangled with the new single-layer config and exposure.
+
+**2. Permanent fix / simplification pros and cons**
+
+Pros of collapsing to a single nginx layer on .30 (NAT public 443 straight to .30, terminate TLS there, proxy `/ws` directly to 127.0.0.1:3001):
+- Removes one entire place where server→agent Text frames can be buffered, delayed, or mishandled. The explicit `proxy_buffering off;`, `proxy_http_version 1.1;`, Upgrade/Connection headers, and appropriate read/write timeouts can be put in one location block and kept consistent.
+- Simpler mental model and fewer moving parts for the agent control path (grey-cloud direct path especially).
+- Slightly lower latency and one fewer internal TCP hop.
+- If the current NPM-generated config has been quietly suboptimal for long-lived bidirectional WS with server-initiated data frames, this eliminates that class of problem permanently.
+
+Cons and real risks:
+- **Exposure change**: .30 becomes directly internet-facing on 443 for rmm-api (and likely the dashboard if the same public IP is used). Today the public face is NPM on .20. You are moving the edge TLS terminator and the first point of public contact onto the origin box.
+- **Cert and renewal problem**: NPM currently holds and auto-renews the LE certs. Moving termination to .30 requires either (a) installing the certs + private keys on .30 and setting up independent renewal (certbot or equivalent) for both subdomains, or (b) some other sharing mechanism. Renewal failure or a botched cutover risks mass agent disconnects (all 200) plus dashboard outage. This is a classic "moved the certs, forgot the renewal" regression vector.
+- **Dashboard entanglement**: Both subdomains currently resolve to the same public IP. Changing the DNAT target for 443 moves *all* traffic (Cloudflare-proxied dashboard + direct grey-cloud API) to .30. You cannot easily move only the agent path without either a second public IP, port tricks (not viable for wss), or careful SNI/Host-based server blocks on .30 that correctly handle Cloudflare's X-Forwarded-For / CF-Connecting-IP plus real-ip module configuration. Keeping the dashboard on NPM while moving only API traffic requires more routing/NAT cleverness than the proposal states.
+- **Cloudflare-specific config**: The orange-cloud dashboard path expects proper trusted-proxy handling, real IP restoration, and possibly HTTP/2 or early hints behavior. Origin nginx on .30 was previously only doing internal plaintext proxy_pass; it will now need the full hardened public + Cloudflare config that NPM was providing.
+- **Grey-cloud direct surface**: rmm-api is already DNS-only grey-cloud, so the public IP was already reachable. But the actual listener and TLS stack were on NPM. Moving the listener changes the attack surface and the exact software that faces the internet for agent connections.
+- **Operational blast radius**: A misconfig on .30's nginx (wrong `server_name`, missing WebSocket upgrade handling, wrong `listen` directives, http2 vs http1.1 issues, etc.) takes down or degrades the entire agent fleet plus the dashboard in one shot. Current setup has some separation (NPM as a distinct TLS/proxy tier).
+- **Rollback complexity**: Re-pointing the DNAT back to .20 and restoring NPM as terminator is possible but not instantaneous, and any state (long-lived WS connections) will be torn down.
+- Minor: HTTP/2 handling for the dashboard, HSTS, OCSP stapling, cipher suite parity, etc., all have to be re-validated on the new terminator.
+
+The simplification is attractive for the agent WS path specifically, but the proposal as written does not isolate that path cleanly from the dashboard and does not address the cert-renewal and exposure shift.
+
+**3. Lower-risk / faster ways to get equivalent diagnostic signal**
+
+Yes, several that are strictly safer than re-NATing production:
+
+- **Highest value, lowest risk first**: Add the missing WebSocket hardening to the *existing* layers before touching routing. On the origin nginx (.30), for the `/ws` location (and any upstream for the agent path), explicitly set:
+  ```
+  proxy_buffering off;
+  proxy_http_version 1.1;
+  proxy_set_header Upgrade $http_upgrade;
+  proxy_set_header Connection "upgrade";
+  proxy_read_timeout 3600s;   # or similar long value
+  proxy_send_timeout 3600s;
+  ```
+  Do the same in NPM's generated config for the rmm-api vhost if it is not already perfect. Reload (not restart) and immediately test command delivery to the known-bad agents. This directly attacks H1 with almost zero blast radius. If commands start flowing, you have strong evidence without any topology change.
+- Run the packet captures that were just installed. Capture on .30 (both the external-facing interface and the loopback to 3001) while deliberately sending a command (small and large) to a bad agent and a good agent. See whether the Text frames ever arrive at .30's nginx listen socket, whether they are forwarded to the Rust process, and whether ACKs or responses come back. Also capture on NPM (.20) if feasible. This localizes where the frame dies without any config or routing change.
+- Server-side instrumentation: add targeted logging (or a temporary debug path) in the Rust server that logs when it attempts to send a command frame to specific agent IDs (or connection IDs) and whether the WS write succeeds or returns an error. Correlate with the agents that are "healthy" by last_seen/heartbeat but never ACK.
+- Controlled client test: from a machine behind one of the affected UDRs (or, if possible, temporarily on the same LAN as an affected agent), use `wscat`, `websocat`, or a tiny test binary to connect to `wss://rmm-api.azcomputerguru.com/ws` and have the server send Text frames. Observe whether this test client receives them. This bypasses the specific agent binary while using the real path.
+- Per-agent or per-site config override if the agent supports an alternate server URL (or can be pointed via a short-lived DNS change + test hostname). Point one affected agent (or a spare test agent placed behind the same UDR) at a bypass that avoids NPM entirely — for example, a temporary high-port listener on .30 with its own TLS (self-signed is acceptable for one test agent) or a minimal nginx instance on .30 listening on an alternate port that you DNAT only for a test public port. This gives a "same UDR, different proxy path" signal with tiny scope.
+- If the agents expose any local diagnostic or can have a one-off config pushed, collect WS receive logs from one bad agent and one good agent during a forced command.
+
+These give you localization (where the frame stops) and a direct test of the "remove the second nginx layer" hypothesis without moving the production NAT target or the TLS certs for 200 agents.
+
+**4. Net recommendation and concrete ordered action list**
+
+Do **not** do the NAT-straight-to-.30 change as the first step. It is high-risk, has significant confounds for the diagnostic, forces a cert/renewal/Cloudflare transition, and likely pulls the dashboard along whether intended or not. It is a reasonable *later* simplification once you have evidence that the dual-layer path is the culprit, but it is not a cheap or clean bisection today.
+
+Ordered action list (do in this sequence):
+
+1. Immediately harden both existing nginx layers for the `/ws` location (add `proxy_buffering off;`, the Upgrade/Connection headers, long timeouts, and `proxy_http_version 1.1;`). Reload and re-test command delivery to the known-bad agents (PST-SERVER etc.). This is the cheapest, highest-signal step for H1.
+2. Run the packet captures on .30 (and .20 if possible) during deliberate command sends to bad vs good agents. Determine whether Text frames reach .30's nginx and whether they are proxied to the Rust process.
+3. Add minimal server-side send logging in the Rust binary for the affected agent IDs so you can see whether the server believes it is writing the frames successfully.
+4. Perform a narrow-path test with one affected agent or a test client (alternate listener on .30, test hostname, or hosts-level override where feasible) that bypasses NPM while keeping everything else identical. This is the closest practical approximation to the operator's proposal without touching the production DNAT or certs for the fleet.
+5. Only if steps 1–4 still point at the dual-layer path (and the captures/logs show frames dying in NPM or the origin nginx), *then* plan the single-layer cutover. At that point treat it as a deliberate simplification project: separate public IP or very careful server-block design if you want to keep dashboard on NPM; explicit cert migration + renewal automation on .30; staged rollout (small cohort first); documented rollback; and full validation of Cloudflare + grey-cloud paths.
+
+The operator's proposal does not cleanly isolate "NPM" because it also moves TLS termination and (almost certainly) forces the dashboard onto the new terminator. The subset nature of the affected agents plus the fact that reconnects already failed already suggest the root cause may be stateful/per-connection rather than a blanket "any traffic through NPM is broken." Get the direct evidence from config hardening + captures + narrow bypass before you change the public routing and cert story for the whole system.

tmp/r4-gemini.txt

@@ -0,0 +1,20 @@
+[OK] [VERDICT] The failure is localized to the **NPM (.20) -> Agent** segment. The origin-to-NPM hop is healthy.
+
+### 1. Updated Hypotheses
+*   **H1 (Top): NPM Connection-Specific Failure / State Corruption.** (Confidence: High). Since ~40 agents work, the "relay logic" isn't globally broken. However, NPM/OpenResty may have stalled the specific TCP/WS streams for these agents (e.g., full send-buffers, socket-state mismatch, or worker-process hang). The fact that the frame reaches NPM but doesn't reach the agent suggests NPM accepted the frame from origin but failed to "push" it to the downstream socket.
+*   **H2: WAN / MTU / Black-Hole (H3).** (Confidence: Medium). A specific path MTU issue or firewall "stale entry" at the UniFi UDR (PST site) could be black-holing downstream packets that exceed a certain size or signature, while allowing the upstream heartbeats.
+*   **H3: Agent-side Socket Freeze.** (Confidence: Low). If the agent OS/NIC was dropping packets, we would likely see the agent trip its 90s timeout.
+
+**Implication of "Subset of Agents":** This points away from global NPM config and toward **per-connection state**. NPM might be holding "ghost" connections or failing to drain buffers for specific client IPs.
+
+### 2. The Decisive Next Action: Packet Capture on NPM (.20)
+**Action:** `tcpdump` on NPM's LAN interface, filtering for the **Agent's Public IP** (98.190.129.150) while sending a command.
+
+*   **Outcome A (NPM emits nothing):** If NPM receives the plaintext frame from .30 but emits **zero** TLS application data toward the agent IP, **NPM is the culprit.** (Logic: The proxy is swallowing the frame internally).
+*   **Outcome B (NPM emits, no ACK):** If NPM emits TLS packets but sees **TCP Retransmissions**, the **Internet/UDR** is black-holing the traffic.
+*   **Outcome C (NPM emits, Agent ACKs, still no execution):** The **Agent** is receiving the data but the WS stack inside the agent is dead/frozen.
+
+### 3. Verdict on "Remove NPM" Idea
+The "Remove NPM" idea is now **highly justified** as a diagnostic isolation step. Since we have provably tracked the packet *to* NPM's doorstep, bypassing it eliminates the most complex variable (TLS termination and WS proxying).
+
+**Cheaper alternative first:** Before a full NAT bypass, **force-restart the NPM Docker container.** If the "stuck" agents immediately ACK commands upon reconnecting, you have confirmed an NPM-internal state corruption (H1) without changing infrastructure. If a restart doesn't fix it, proceed immediately to the NAT bypass (Direct to .30).

tmp/r4-grok.txt

@@ -0,0 +1,40 @@
+**1. Updated ranked hypotheses**
+
+The new evidence (commands provably leave Rust, leave origin nginx .30, and are sent toward NPM .20) eliminates everything before NPM. The fact that ~40 other agents on the identical NPM path continue to receive commands normally is the strongest signal.
+
+- **NPM buffering/relay defect** (per-connection or per-stream): Still plausible but now narrowed. Global NPM breakage is refuted. A defect that only affects a subset of long-lived WS connections (e.g., specific idle-timeout / buffer / upgrade handling / connection-tracking state inside the nginx instance inside the Docker container, or only after certain connection age or header combinations from particular agents) remains possible. The 13x vs 8x marker count hint of retransmit/re-offer on the internal leg is consistent with the proxy sometimes being slow or lossy on the client leg for those particular sockets.
+
+- **NPM->WAN/UDR black-hole (H3)**: Remains high. Different affected sites (PST UDR + GTS-PEDRO-H) make a single UDR bug less likely, but per-agent-path problems are common: UniFi stateful inspection / NAT table quirks on inbound WS data frames (even when outbound heartbeats keep the flow alive), silent drops after the TCP connection is established, MTU/PMTUD issues on the return path for some ISPs, or middlebox behavior that treats small outbound heartbeats differently from inbound application data. Tiny 80 B frames failing is compatible if the drop is flow- or direction-based rather than size-based.
+
+- **Agent-side defect (H2)**: Now the strongest single hypothesis for the subset pattern. The WS connection is clearly alive from the agent's viewpoint (it emits heartbeats, does not hit the 90 s no-inbound reconnect timer). This is exactly what you expect if the agent process is still writing on the socket and the TCP stack is ACKing, but the receive side of the agent either (a) is not reading the socket for "command" frames, (b) has a dispatcher that drops or mishandles "type":"command" messages while still handling pings/heartbeats, or (c) is under local resource pressure / wedged thread only on those specific hosts. Same agent version + only a subset affected fits an agent-local trigger (particular host load, Windows firewall / security product interaction with the socket, specific network interface, etc.). The two PST servers being at the same site is consistent with them sharing the same local conditions.
+
+"Subset on the same NPM" therefore pushes weight toward either per-connection state inside NPM or (more likely) causes that are downstream of NPM and specific to the individual agent or its access path. It does not favor a broad NPM configuration or capacity problem.
+
+**2. Single most decisive next test**
+
+Targeted packet capture on the NPM box (.20) itself, focused on the established TCP 4-tuples for the affected agents (source ports on 98.190.129.150 etc.), while sending additional uniquely marked commands. Capture at the host level (or the relevant Docker bridge) so you see what actually leaves the NPM container toward the public internet. Also capture the corresponding inbound heartbeats/ACKs on the same flows. You already have the marking technique that makes correlation trivial.
+
+This is decisive; the three clean outcomes map directly to the three remaining locations:
+
+- No TCP data / TLS application_data packets emitted from .20 toward the agent IP shortly after the frame is seen arriving at NPM on the internal leg → NPM received the WS message but did not forward it on the client connection. → NPM buffering/relay defect (or its per-connection state for that socket).
+
+- Packets are emitted promptly from .20, normal TCP ACKs come back from the agent IP (often within one RTT), but the agent still never ACKs/executes the marked command → the bytes reached the agent's OS, were delivered to the TCP socket, and were ACKed. The defect is after TCP delivery inside the agent process. → Agent-side defect (H2).
+
+- Packets are emitted from .20 but you see repeated retransmissions (RTOs), growing send buffer, zero-window probes, or missing ACKs for NPM's data segments, while the agent continues to send heartbeats that NPM is ACKing → data is leaving the NPM host but is not reaching (or not being accepted by) the agent's TCP stack. → Black-hole or severe loss on the NPM→agent leg (H3 or UDR / path / host firewall).
+
+Secondary signals (ICMP unreachable, RSTs from intermediate hops, TCP timestamps, etc.) can further localize inside the path.
+
+Is the .20 capture decisive enough? Yes, for practical purposes. It directly answers "did NPM emit?" and "did the far end TCP-ACK the data?" without requiring any agent reconfiguration. It is low-risk, quick to set up, and can be done while the system is live.
+
+Better move? Only marginally: first do a quick non-capture check on .20 (NPM container logs + access/error logs for the exact vhost and the affected connection IDs around the mark timestamps; `ss -tpi` or `ss -tinfo` on the sockets to see send-q, last-ack, etc.). If logs are silent and sockets look healthy, proceed straight to the capture. Do not yet touch NPM advanced config or origin nginx unless the capture implicates the proxy.
+
+**3. Verdict on "remove NPM / NAT straight to .30"**
+
+The new evidence does make the bypass more justified as a *targeted follow-up*, because everything before NPM is now exonerated. If the capture shows NPM is not emitting the frames, the bypass becomes high-confidence: you would be removing the exact component that is failing to relay.
+
+However, the capture is still the cheaper, safer, higher-ROI step first. Reasons:
+- It isolates NPM itself with almost no blast radius and no production routing change.
+- If the capture instead shows NPM *is* emitting + retransmits or clean ACKs, the bypass is unlikely to help. The packets would still have to traverse the public internet from the same public IP (72.194.62.10) to the same agent WAN IPs; only the internal origin changes from .20 to .30. The black-hole or agent receive problem would remain.
+- Bypass requires edge firewall DNAT change + moving TLS termination to .30 (or reconfiguring the origin nginx) + potential SNAT/return-path adjustments. That carries risk to the ~40 working agents and is harder to roll back cleanly.
+
+Do the .20 capture (with the marked commands) first. Use its outcome to decide whether the bypass is worth the cost and risk, or whether you need to move to agent-side investigation (on-site, reinstall, debug logging in the agent, etc.) or path debugging at the affected sites' UDRs. The capture gives you the information needed to avoid an expensive test that may not be diagnostic.

tmp/r5-gemini.txt

@@ -0,0 +1,47 @@
+[INFO] ROUND 5 DIAGNOSIS
+
+# Verdict: NPM state/buffer exhaustion (H1) is the primary suspect. The "Three-Site" evidence effectively refutes Site-Gateway (H3) as a root cause.
+
+## 1. Hypothesis Re-ranking & Shared Attribute Analysis
+The fact that three different ISPs and physical sites (Site A, B, C) exhibit the same failure signature—specifically the loss of server->agent Text frames while pings persist—makes a "network path" issue (H3) statistically improbable.
+
+**Re-ranked Hypotheses:**
+1.  **[SUSPECT] H1: NPM/Nginx Proxy State.** A specific interaction between Nginx's `proxy_buffering` and the WebSocket tunnel, possibly exacerbated by `http2` protocol-switching overhead or "orphan" buffers in the Docker bridge.
+2.  **[SUSPECT] H2: Agent/Host Environment.** Shared attribute: **Windows Server OS and/or NIC Offloading.**
+    *   *Reasoning:* "PST-SERVER" and "PST-SERVER2" are explicit. If GTS-PEDRO-H is also a Server OS, the shared factor is likely the **Windows TCP Stack** (specifically Receive Window Auto-Tuning or Chimney Offload) reacting to NPM's MTU/buffer handling.
+3.  **[REFUTED] H3: Site Gateway.** Too many variables (different ISPs/UDRs) for a synchronized failure.
+
+**Shared Attribute Candidate:** Long-lived connection age. If these 3 agents have the highest uptime/connection duration, they may be the only ones that have hit a specific Nginx `proxy_read_timeout` or `keepalive` edge case where the socket is "half-closed" in the proxy's eyes but pings (which are small/control frames) still sneak through.
+
+## 2. HTTP/2 and Proxy Buffering Interaction
+`http2 on` combined with `proxy_buffering on` (default) is a known "danger zone" for WebSockets in Nginx.
+*   **The Problem:** While Nginx *should* disable buffering for upgraded connections, the presence of `proxy_buffering on` can cause Nginx to attempt to buffer "large" frames into temporary files if it perceives the client (agent) as slow.
+*   **The Failure:** If NPM is using `http2` toward the agent, it is multiplexing the WS stream. If the "Text frame" (command) is large or follows a specific pattern, Nginx might be "waiting" to fill a buffer before flushing the H2 stream, whereas tiny Ping/Pong frames bypass this or fit in the first chunk.
+*   **Verdict:** This perfectly explains why "Text frames leave origin but never reach agent."
+
+## 3. Recommended Sequence (Rounds 6-7)
+
+**Step 1: Configuration Hardening (Information Gained: High | Risk: Low)**
+Modify the NPM Proxy Host config for the RMM entries to include:
+```nginx
+proxy_buffering off;
+proxy_read_timeout 3600s;
+proxy_send_timeout 3600s;
+```
+*Rationale:* Forcing `proxy_buffering off` is the industry standard for stable WebSockets. It prevents Nginx from sitting on downstream data. A reload is non-destructive.
+
+**Step 2: Socket State Inspection (Information Gained: High | Risk: None)**
+Run `ss -tpni` on the NPM host (.20) and filter for the IP of PST-SERVER (`98.190.129.150`).
+*Look for:* `unacked` bytes or a collapsed `rcv_space`. If `unacked` is high on the NPM->Agent side, the proxy is trying to send, but the agent's TCP window is closed.
+
+**Step 3: Protocol Downgrade (Information Gained: Medium | Risk: Low)**
+Disable `http2` in the NPM UI for this host.
+*Rationale:* Eliminates the H2-to-H1.1 translation layer which is often where "frame-stuck" bugs live in Nginx.
+
+## 4. The NPM Restart Probe
+**Restarting the NPM container is a high-value one-shot probe.**
+
+*   **Result A (Recovery):** If agents recover, it proves **State Corruption** in the NPM/Docker bridge (e.g., conntrack table exhaustion or a hung Nginx worker process buffer).
+*   **Result B (No Recovery):** If they reconnect and *still* fail commands, it proves a **Protocol/Configuration Incompatibility** (e.g., the way NPM 2.x handles WS frames is permanently incompatible with the Windows Server TCP stack or the Agent's WS implementation).
+
+**Recommendation:** Perform Step 1 (Config tweak) *first*. If that fails, Restart. Config tweaks survive restarts; restarts alone only clear symptoms.

tmp/r5-grok.txt

@@ -0,0 +1,74 @@
+**1. Re-rank and shared attribute**
+
+Per-site-gateway (H3) drops sharply. Three independent ISPs, three distinct public IPs, two explicitly UniFi UDR and one different hardware/ISP, all exhibiting the *exact* same signature (heartbeats + agent→server frames work; server→agent Text/command frames disappear after the origin nginx hands them to NPM) is extremely unlikely to be three independent gateway black-holes or asymmetric-routing failures. A single-gateway bug would have to be replicated across vendors or would require the gateways to all react identically to some property of packets sourced from the single public IP of the RMM host — possible in theory (e.g., all mishandle a particular TCP option, window scale, or large frame) but now a low-probability explanation.
+
+Agent-side (H2) and NPM-relay/state rise. The common element is no longer "the network path" but "these three long-lived connections are currently terminated at this specific NPM instance and are being handled by the agents running on these three machines." Global NPM and global 0.6.63 are already falsified by the 40 working agents.
+
+**Most likely shared attribute class (concrete + falsifiable):**
+
+- **Windows Server OS vs workstation OS (or "server role" detection).**  
+  The hostnames (PST-SERVER, PST-SERVER2, GTS-PEDRO-H) are suggestive. The 40 working agents are probably Win10/11 workstations/laptops that sleep, reboot, or lose network frequently. The three affected are almost certainly always-on server hardware running Windows Server (or treated as servers). This is easily falsified: query the agent-reported `os_version`, `platform`, `product_name`, or any "is_server" / role field for exactly these three rows versus a sample of 10–15 working agents. Also check reported hostname patterns and any installer "server" branch.
+
+- **Connection age / long-lived without reconnect (second strongest).**  
+  Servers rarely sleep or reboot, so their WS connections can be days or weeks old. Workstations cycle connections on wake/suspend. A resource leak, buffer state, or internal nginx fd event-loop entry that only manifests after N hours/days of a single upgraded connection would hit precisely the always-on machines. Falsifiable: pull the `connected_at`, `last_heartbeat_at`, or equivalent session start timestamp (or derive from metrics/logs) for the three vs. others. Also look at any "reconnect count" or "uptime" the agents report.
+
+- **Enrollment/update cohort or config snapshot at connection time.**  
+  Less likely but cheap to check: same narrow enrollment window, same build hash + config version at first connect, or a one-time policy push that only these three received before the population diverged. Query enrollment timestamp, last update timestamp, and any per-agent policy or "features" blob.
+
+- **Server-grade NIC / TCP offload / RSS / checksum offload.**  
+  Common on physical servers. These can produce subtly different TCP behavior (TSO, LRO, delayed ACKs, different window scaling) that interacts badly with the return path through Docker bridge + NPM + internet when the proxy is doing WS frame forwarding. Harder to observe remotely but can be inferred if the OS check points to servers and packet captures later show retransmit or zero-window behavior only on these three.
+
+- Weaker: pure idle pattern (servers send almost no other traffic) or specific outbound NAT characteristics at the three sites. These are possible but secondary to the OS/role + age hypotheses.
+
+The pattern "these three but not the other ~40 on identical infrastructure and version" is the signature of either (a) a property of the *machines themselves* (OS, hardware, role, uptime) or (b) a per-connection state that only these three conns accumulated inside NPM.
+
+**2. http2 + proxy_buffering + WS upgrade downstream stall**
+
+Yes, this combination is a known source of subtle, one-directional problems on long-lived upgraded connections.
+
+- `http2 on` applies to the client→NPM leg. The backend leg is forced to 1.1 with the explicit `proxy_http_version 1.1` + Upgrade/Connection headers. This is the standard and usually correct way to do WS through a terminating proxy.
+- However, `proxy_buffering on` (the nginx default, and not overridden) still influences response handling. For a 101 upgrade, nginx is *supposed* to stop using the response buffer and pass data bidirectionally on the raw sockets. In practice, across multiple nginx versions (especially 1.20–1.24 range that NPM often ships), there are edge cases:
+  - Large or bursty server→client Text/Binary frames can still trigger internal temporary buffering or be subject to the proxy's send buffer limits.
+  - The "an upstream response is buffered to a temporary file" warnings are only logged for non-upgraded responses; WS frames can stall silently.
+  - `proxy_read_timeout 60s` (default) governs the upstream read. If the origin is idle, this can close the upstream side even while the client→server direction (heartbeats) keeps the downstream leg alive from the client's perspective. The result is exactly the observed symptom: client can still send (heartbeats reach Rust), but server frames stop being delivered.
+  - Docker bridge + NAT on the client side of NPM adds another layer of buffering and potential delayed ACK or window interaction.
+- There are documented nginx + WS + http2 issues (and NPM-specific reports) where one direction (commonly server→client) stops flowing while pings/heartbeats continue, especially on connections that have been upgraded for a long time or when the client read side is slow. Adding `proxy_buffering off;`, `proxy_read_timeout 300s` or higher, and sometimes `proxy_send_timeout` is the standard mitigation.
+
+The absence of WS-specific errors in the NPM log is consistent with a silent stall rather than a hard failure. The fact that only three connections are affected is also consistent: most of the other 40 either reconnected more recently (fresh state) or are less idle (different buffering/timeout trigger points).
+
+**3. Most discriminating test sequence (next 2–3 rounds), ordered by information gained per unit of risk**
+
+Risk here is primarily "disruption to the ~200 agents" (NPM restart or reload) and secondarily "time to next data point." We have passive/low-risk moves that directly attack the "what is special about these three" and "is data even leaving NPM" questions.
+
+**Round 5–6 (near-zero risk, highest immediate value):**
+- Pull the agent DB rows (or equivalent telemetry) for the three affected + a representative sample of working agents on the same NPM/0.6.63. Extract: OS/version/platform strings, hostname, enrollment/first-seen, connection/session start time or age, last reconnect, any policy/site/customer tag, reported uptime or last reboot, any capability or config hash differences. This is the single highest-leverage action right now — it will either confirm or kill the Windows Server + long-lived-conn hypotheses within minutes.
+- While that query runs, complete/analyze the in-progress NPM-box capture of the NPM→agent leg for the three IPs. Correlate exact timestamps of a deliberately sent test command (Text frame) with what appears on the wire.
+- On the .20 host (outside the container), run `ss -tpi` / `ss -tan` (and `conntrack` if available) filtered to the three public IPs. Look at send-q/recv-q sizes, timer state, retransmit counts, window sizes right after a test command is injected. Also capture (tcpdump/wireshark) on the host's public interface for traffic to those three destinations. This tells us immediately whether the frames are leaving the host at all, whether they are retransmitting, or whether the sockets look "healthy but idle."
+
+**Round 6–7 (low-to-medium risk, targeted config change):**
+- Edit the NPM proxy host config for the rmm-api location: add `proxy_buffering off;`, raise `proxy_read_timeout 300s;` (or 600s), and `proxy_send_timeout 300s;` if available. Reload nginx inside the container (`nginx -s reload`). Do **not** restart the whole container yet.
+- Immediately after reload, send test commands to the three affected agents and watch both the new host capture and the agent side (CommandAck / execution).
+- This directly tests the known nginx WS + buffering + timeout failure mode without a full reconnect storm. If the three start working and no others regress, we have a strong causal signal and a fix. If nothing changes, we have ruled out the default buffering/timeout path as the sole cause for these conns.
+
+**Round 7+ (higher risk, one-shot probe):**
+- Only after the above: consider a deliberate NPM container restart. See question 4 for interpretation.
+
+Secondary cheap actions that can run in parallel: inspect NPM access/error logs for bytes-written on the three specific connections, look for any conntrack table oddities or Docker network restarts around the time the problem appeared, and check whether the three agents show any difference in ping/pong timing or last successful upstream frame.
+
+**4. Restarting the NPM container as a probe**
+
+It is a worthwhile one-shot probe, but it should be the *last* move in the sequence above, not the first, because the cost is real (brief loss of command delivery for the entire fleet while ~200 agents reconnect).
+
+Interpretation is clean and high-value:
+
+- **Stuck agents recover (they begin receiving Text frames and acking/executing after the reconnect)**: Strong evidence that the defect was in *per-connection state inside the running NPM/nginx process*. Classic examples: a particular upgraded socket's event loop entry got into a bad state, an internal send buffer or WS frame queue for that fd was wedged, or a long-idle connection accumulated a condition that only a fresh accept + upgrade cleared. The fact that only three were affected fits a rare per-conn corruption or timeout path. This also exonerates (or greatly weakens) "the agent machines themselves are permanently broken" and most per-site gateway theories, because the new connections from the same three public IPs now work.
+
+- **Stuck agents still do not receive commands after restart (while the rest of the fleet does)**: The problem is *not* transient NPM internal state. This shifts weight heavily to:
+  - Agent-side (H2): the three agents, once they re-establish, still cannot process inbound frames (possible causes: OS-level socket receive handling, WS client library state machine bug triggered by something in their environment, server-grade NIC offload interacting badly with the specific return traffic pattern, or the agent process having a stuck read goroutine / select that only affects certain machines).
+  - Or a persistent property of the return path from this specific public IP to those three destinations (e.g., the three gateways all apply some policy or have a path MTU / TCP option incompatibility that only manifests for certain frame sizes or after the initial upgrade handshake). Three different ISPs still makes this surprising, but a restart would have forced fresh TCP connections, so any "established conn half-open" theory would be weakened.
+  - Less likely: something in the origin nginx or Rust side that only these three agent identities trigger.
+
+A restart also gives a natural before/after on connection age: if the problem reappears only after the new connections have been up for many hours/days, that further supports the long-lived-conn hypothesis.
+
+**Summary recommendation for the next round**
+Do the DB attribute query + host ss + targeted capture first (today). Apply the `proxy_buffering off` + longer timeouts + reload next (cheap and directly addresses a documented class of nginx WS failure). Only then decide on a container restart. The three-different-ISPs fact has already done most of the work of demoting H3; the population difference is now the best remaining signal, and the low-risk data pulls will likely tell us whether we are looking at "server OS + long conn age" or "rare NPM per-conn stall."

tmp/r6-gemini.txt

@@ -0,0 +1,38 @@
+[INFO] VERDICT: The contradiction is an illusion caused by assuming the agent's read path is monolithic. The read loop is actively pulling bytes (handling Pings), but the command dispatch path has silently failed (likely a panicked task or dropped channel). 
+
+Confidence: High. The combination of TCP ACKs and successful WS Pong responses strictly requires the agent's WebSocket read loop to be actively polling the socket. 
+
+Here is the independent review of your questions:
+
+### 1. Agent-side mechanisms (Ranked)
+[OK] **Rank 1: Panicked/Dead Worker Task (Channel Disconnect).** 
+The WS read loop is actively running (calling `.next()` or `.read()`), which allows the underlying library (e.g., `tungstenite`) to automatically process Pings and send Pongs. When a Text frame arrives, the loop attempts to send it over an internal channel (e.g., `mpsc`) to a command-executor task. If that executor task panicked or died, the channel is closed. The read loop receives a `SendError`, ignores it (or logs it, which we can't see), and loops back to read the next frame.
+[WARN] **Rank 2: Logic Bug / State Desync in 0.6.63 Router.** 
+The agent reads the Text frame, but an internal state machine believes it is not authenticated or not ready, and silently drops the frame.
+[ERROR] **Rank 3: TLS/WS Reassembly Desync.** 
+Refuted. A desync at the TLS or WS frame level would corrupt the stream, causing protocol errors and connection termination. It would not cleanly drop Text frames while perfectly parsing Ping frames.
+
+### 2. Is "kernel ACKs but app stopped reading" plausible?
+[ERROR] VERDICT: REFUTED. 
+If the application had completely stopped reading (e.g., blocked thread or deadlocked read loop), the kernel would ACK packets until the TCP receive window filled up (zero-window). **However, this cannot explain the Pings.** Pings are WebSocket control frames inside the TLS tunnel. For the server to receive Pongs, the agent application *must* be continuously decrypting TLS records and parsing WS frames. The application has not stopped reading; it is selectively ignoring or failing to process the payload of the Text frames.
+
+### 3. Non-Agent Explanations (Middleboxes, Offload)
+[WARN] **TLS Inspection / Deep Packet Inspection (Middlebox):**
+Plausible, but unlikely. If the agent's site uses a corporate firewall with TLS MITM (terminating TLS with a trusted root cert), the firewall could theoretically parse WS frames and drop Text payloads it deems malicious (e.g., matching a C2 signature) while passing Pings. 
+[ERROR] **GRO/LRO/TSO Offload Corruption:**
+Refuted. Hardware offload corrupts IP/TCP headers or checksums, leading to packet drops at the kernel level. It does not perfectly reassemble TLS, parse WS framing, and selectively drop application Text frames.
+[ERROR] **TCP Proxy without MITM:**
+Refuted. A transparent TCP proxy might ACK the packet early, but if it fails to deliver the payload to the agent, the agent's WS stream would eventually stall, and Pings would stop working.
+
+### 4. The Single Most Decisive Next Test
+[INFO] VERDICT: **Server-side connection eviction (Force disconnect).**
+Kill the specific TCP connection for one of the affected agents (e.g., drop it at NPM, or push a disconnect command from the Rust server's internal state to force a socket close). 
+
+**Reasoning:** 
+This forces the agent to establish a completely fresh TCP/TLS/WS session and spawns fresh asynchronous tasks on the agent side. 
+*   If commands **succeed** immediately after reconnect: It definitively proves the issue is a transient state corruption, deadlock, or panicked task inside the agent software (0.6.63) that accumulates over time.
+*   If commands **still fail** (but pings work): It heavily points to a persistent middlebox/firewall dropping the payload, or a deterministic bug triggered immediately upon connection in 0.6.63.
+
+### 5. Significance of "worked at 22:23 right after update, never since"
+[INFO] VERDICT: This strongly supports the "panicked worker task" or "deadlock" hypothesis. 
+The agent successfully connected, initialized, and processed exactly *one* command. The processing of that first command (or a background task spawned shortly after, like the log-uploader) likely triggered a panic or a permanent channel deadlock in the new 0.6.63 codebase. Once that specific task died, the read loop remained alive (answering Pings), but all subsequent Text frames routed to that dead task are now being dropped into the void.

tmp/r6-grok.txt

@@ -0,0 +1,53 @@
+**1. Agent-side mechanisms that ACK TCP + handle Ping but starve Text frames (ranked by plausibility given the evidence)**
+
+The capture + ongoing pings + outbound heartbeats already prove the TCP bytes arrived, the TLS records were decrypted, and the WS frame reader advanced past the 193-byte Text payloads (TCP is a strict ordered byte stream; you cannot reach a later Ping in the stream without having consumed the preceding command bytes in userspace).
+
+Highest to lowest:
+
+- **Command dispatch / executor side is not consuming the parsed Text messages** (most likely). The low-level frame loop (or auto-control path) still runs, sees Ping, emits Pong, and keeps the socket moving. The Text is parsed but handed to a channel / task / match arm that has no active consumer, is parked, or was disabled after the first post-update command. Classic patterns: `tokio::spawn`ed handler that deadlocked on a shared resource/mutex after the successful 22:23 command; mpsc receiver with no task draining it; select! loop where the "commands" branch is no longer selected after a state transition.
+- **Per-session state machine / phase / "current command" slot left in a terminal state after the first success**. The 22:23 command (or the ConfigUpdate + inventory that preceded it) advanced a flag, consumed a oneshot, set a "busy" or "awaiting result" guard, or switched the agent into a mode that drops or ignores subsequent regular command Text frames. Later pings still flow because they are handled in the control path before or outside that state machine.
+- **High-level message stream vs. control frame path divergence in the WS library**. The app code only polls a `Stream` of user messages (Text/Binary) that is no longer being driven (or whose internal buffer is not drained), while the library (or a small helper) keeps reading the underlying TLS stream specifically to service Ping/Pong/Close automatically. This is common in split read-half + auto-pong designs.
+- **Silent drop after parse but before ack path** (e.g. command ID dedup, unknown opcode after reassembly edge, payload that no longer matches an expected envelope post-update, or write-half for acks is on a different handle that is stalled while heartbeat writes use a timer path that still works).
+- **Internal WS reassembly buffer desync that only affects data frames**. Less likely: control frames are short and non-fragmented and can be processed even when a data message is in "waiting for continuation" state. The fact that tiny commands worked once and pings keep working makes a persistent parser state corruption less probable than an app-level dispatch problem.
+- Pure "recv buffer filled, kernel still ACKing, app not reading at all" is ruled out by the later pings (see #2).
+
+**2. "Kernel ACKs while app stopped reading" plausibility and ping reconciliation**
+
+It is the normal TCP behavior: the kernel ACKs as soon as segments land in the socket receive buffer; userspace `read()` / TLS decrypt / WS parse is completely independent. So the three 214-byte frames being ACKed with no retransmit only tells us the bytes reached the agent's TCP stack.
+
+It cannot explain the observed pings, however. Because the pings (31-byte TLS frames) were interleaved or arrived after the commands in the TCP byte stream, any userspace code that saw and responded to a later Ping necessarily performed enough reads to drain past the command bytes. Therefore the WS frame parser in the agent did see the Text frames. The break is after parsing — in dispatch, state, or the consumer of user messages. If the pings that were answered were strictly the ones before the commands in the stream and no further pings arrived afterward, this would be weaker, but "continues to answer" and "interleaved" make the strict "app has stopped reading the socket" story untenable.
+
+**3. Non-agent explanations (middlebox, UDR inspection, offload) weighed**
+
+- **Agent-site middlebox / transparent TCP proxy / UDR doing L4 mangling**: Low. The capture on the NPM side sees the TCP 4-tuple terminating at the agents' own public IPs (98.190..., 64.139..., 68.230...) with clean ACKs coming back from those same IPs. A real proxy or interception device would normally terminate the connection and present its own source IP to NPM for the server leg (or would require a transparent L3 forwarder that does almost nothing to the TLS/WS bytes). 
+- **TLS inspection / MITM at the agent's UDR or local gateway**: Very low for the same reason. Full MITM terminates TLS on the middlebox, re-encrypts toward NPM, and the source IP NPM sees becomes the middlebox egress IP, not the listed agent public IPs. The agents would also have to trust the middlebox CA; a Rust agent using system roots or rustls is unlikely to do so silently unless the machines are heavily managed. Content-aware mangling inside TLS is not possible without breaking TLS.
+- **GRO/LRO/TSO offload corruption on the server-grade NICs**: Possible in principle but low weight here. These offloads sit below TCP; corruption that still produces valid TCP checksums and clean ACKs with no retransmits would be a very specific driver/firmware bug. It would also tend to affect the entire stream (pings and commands are just TLS records over the same TCP bytes), not selectively Text frames. The "worked for exactly one command right after update, then never again" pattern is atypical for a hardware offload issue (those are usually consistent from the first packet). Different ISPs make a common NIC/driver across exactly the broken agents less likely unless they are all the same hardware model.
+
+Overall the capture + behavior already pushes probability very heavily onto the agent process after the WS Text is parsed.
+
+**4. Single most decisive next test (given constraints)**
+
+Force-evict the three agents' WebSocket sessions from the server side (allowed). As soon as each shows a fresh connection (DB last_seen / connection event), immediately dispatch one CLEAN test command to it while running a fresh, filtered capture on the NPM host (and ideally also on the origin side) for traffic to/from exactly those three public IPs. Also read the 22:21 last-uploaded log batch for these agents if it is still available in the DB.
+
+Why this single action is best:
+- It directly tests whether the problem is tied to the long-lived session state versus reproducible on every new connection.
+- If commands now produce CommandAck + execution after a fresh connect: the root cause is almost certainly per-connection accumulated state inside the agent (stuck executor, consumed channel, uncleared "busy" flag, reassembly state after the 22:23 command, etc.). This exonerates NPM/pf path further and tells us the bug is in the agent's WS read loop + command dispatch after the first post-reconnect message.
+- If the exact same symptom repeats on the new connection (TCP ACKed, pings answered, no CommandAck, no execution): the problem lives in the agent binary's core message handling for these hosts (or something local and persistent on those three machines). At that point we have strong evidence to deprioritize further NPM/pf changes and instead focus on (a) deep analysis of the 22:21 logs for init differences, (b) any per-agent DB state or config that these three share and others don't, and (c) eventually asking the sites for local captures or offload status (since we cannot push or visit).
+- It is cheap, targeted, does not disturb the other ~40 working agents, and produces a clean before/after capture we can compare to the previous decisive capture.
+- Restarting NPM would also force reconnects but is heavier and less precise.
+
+Secondary cheap adjunct while doing the eviction test: ensure an explicit outbound SNAT rule exists for 172.16.3.20 (even though the prior ACK behavior already argues against the SNAT gap being causal for these flows).
+
+**5. "Worked at 22:23 right after the 0.6.63 update, never since" implications**
+
+Yes, this is highly diagnostic. It almost certainly means:
+- The update caused a process restart / fresh WS connection.
+- The very first command(s) after that fresh connection (ConfigUpdate success + the one at 22:23) exercised a code path that succeeded.
+- Something in the handling of that first command (or the ConfigUpdate + inventory that happened at update time) left the agent's command-receive or command-execution machinery in a broken state for all subsequent commands on that connection.
+- Or the 22:23 command used a different code path (update-time special command, different envelope, different handler registration) than the later "hostname" / CLEAN commands.
+
+This pattern is classic for: one-shot initialization that runs only on first message or right after connect; a channel or oneshot that is consumed once; a state machine that advances to a "post-first-command" mode that has a bug; a resource (lock, slot, pending-command table) acquired by the successful command and never released; or a difference between the commands the updater itself sends versus the ones the server later dispatches.
+
+The fact that only three agents (different ISPs) show it while dozens of other 0.6.63 agents do not suggests these three either (a) hit a narrow race or ordering during their update/reconnect that others missed, (b) have distinct persisted local state or recent command history that interacts with the new version, or (c) share some other per-agent attribute the server or agent code now treats specially.
+
+The decisive reconnect test above will tell us whether a brand-new post-eviction connection reproduces the "only the first command works" behavior or restores normal function. That single bit of information collapses most of the remaining hypotheses.

tmp/r7-gemini.txt

@@ -0,0 +1,24 @@
+# GuruRMM command-delivery diagnosis — ROUND 7 (SNAT test result)
+
+[VERDICT] The root cause is an application-level logic failure or state corruption within the v0.6.63 agent. 
+Confidence: 95%.
+
+[REASONING]
+1. [NETWORK CLOSED] The packet capture confirms Text frames reach the agent; the kernel's TCP ACK confirms delivery. The fact that WS pings (application-level) continue on the same connection proves the WebSocket framing and the agent's main read loop are fully functional.
+2. [STATE SMOKING GUN] The observation that only a process restart (clearing memory/state) allowed a single success—while a WS reconnect (clearing network/proxy state) did not—is the definitive indicator of an internal agent bug.
+3. [PATH ANALYSIS] Since pings work, the code is reaching the `handle_server_message()` dispatch. "No CommandAck" implies the function is exiting (due to a parse error) or blocking (due to a lock/logic error) before the ACK is sent.
+
+[ANSWERS]
+
+1. [SNAT TEST] The SNAT test result, combined with the capture, fully closes the network hypothesis. While pfSense `reply-to` might have bypassed the SNAT rule for existing inbound states, the agent's TCP ACKs prove the return path was already valid. The failure is not in the bytes moving; it is in the agent's processing of those bytes.
+
+2. [FINAL VERDICT] The bug is in the v0.6.63 agent's "Slice B" implementation of `handle_server_message()`. 
+   Most Probable: A `serde_json` deserialization failure caused by a schema mismatch or a malformed `ServerMessage` payload that only triggers under specific conditions (possibly related to the new dedup/results cache fields). Because the error is logged-and-ignored, the loop survives for pings but "swallows" every command.
+
+3. [ACTION PLAN]
+   (a) [RECOVERY] Force-restart the `gururmm-agent` service on the 3 affected machines using an out-of-band tool (e.g., ScreenConnect, local shell, or a separate "emergency" updater if it exists).
+   (b) [AGENT FIX] Audit `handle_server_message` in the agent code. Move the `CommandAck` to the absolute first line after a successful parse. If the parse itself is failing, inspect the raw JSON being sent by the server for hidden characters or missing fields required by the new v0.6.63 struct.
+   (c) [INFRA] Roll back the pfSense SNAT rule. It is confirmed unnecessary and adds "magic" to the routing table that didn't solve the issue.
+   (d) [DEDUP AUDIT] Ensure the dedup cache logic does not return *before* sending a `CommandAck`. Even if a command is a duplicate, the agent MUST ACK to stop the server from retrying.
+
+4. [LOW-RISK TEST] Send a command with a minimal, primitive payload (e.g., `id`) to the affected agents. If it still fails, the `serde_json` schema itself is likely the culprit. If it succeeds, the issue is payload-size or content-specific. Check server logs for any "broken pipe" or "write timeout" errors specifically when sending the larger `Command` Text frames.

tmp/r7-grok.txt

@@ -0,0 +1,55 @@
+**1. Does the SNAT test + capture fully close the network/SNAT/proxy class?**
+
+Yes for practical purposes. The packet capture already showed the Text command frame delivered to the agent's kernel (no retransmits, TCP ACKed by the agent). The same connection continues to answer server pings (bidirectional app-level traffic). The SNAT rule produced zero change in behavior on a freshly negotiated connection.
+
+The reply-to/state bypass caveat is real for classic pfSense inbound DNAT return traffic, but it does not leave a plausible residual network explanation here: the agent is already successfully returning TCP ACKs and pong/ping responses on that exact 5-tuple. If the return source IP or path were unacceptable to the agent or an intermediate, those ACKs and ongoing pings would not work. Proxy/NPM truncation, MTU, or SNAT asymmetry also fail to explain "commands are received and ACKed at TCP but produce zero CommandAck while pings on the identical WS connection are answered."
+
+Network/SNAT/proxy hypothesis class is closed.
+
+**2. Final verdict on root cause and location**
+
+Most probable root cause (confidence ~80): a stateful bug in the v0.6.63 agent (comms-durability slice B) inside `handle_server_message()` / the command arm — specifically the new dedup/results cache + "CommandAck immediately, then spawn" logic.
+
+Signature that only appears after process start: exactly one command succeeds (Ack sent + task spawned), then all subsequent commands on that process (including after WS reconnects) produce no Ack at all, while the read loop continues to handle pings. This is the textbook symptom of per-process in-memory state (cache, a moved channel/sender, a "first command done" guard, a poisoned or never-recreated oneshot, or a dedup key that is not the per-request ID) being clean only at process startup. The first post-restart command "poisons" it for everything after.
+
+Parse failure is also possible but less likely given the ping path stays alive; a silent early return or `if dedup.contains(...) { continue; }` before the Ack line is a stronger fit. The inline read-loop placement plus "Ack first, then spawn" + cache addition makes this the exact place the durability changes could have introduced a one-command-then-dead dispatch path.
+
+Network, NPM, pfSense, origin, and server dispatch are all exonerated by the capture + ping survival + SNAT result.
+
+**3. Concrete recommended action plan**
+
+a) Immediate recovery of the 3 agents (PST-SERVER/PST-SERVER2/GTS-PEDRO-H):
+- Do not use the command channel. Use out-of-band access (console/iLO/iDRAC, local RDP/WinRM, scheduled task, or any pre-existing management path).
+- Preferred: roll back those three hosts to the last known-good agent version (pre-slice-B / pre-0.6.63). A plain process restart will only buy one working command per the observed pattern, which is useless for sustained recovery.
+- If the agent update/installer path is independent of the WS command channel (the fact that the 0.6.63 update itself succeeded suggests it may be), push the rollback package to just those three. Verify with multiple distinct commands after rollback.
+- Note the hosts as "on old comms code until fixed build"; do not leave them on the broken version.
+
+b) Pinning the exact 0.6.63 agent bug (no live access to the broken agents):
+- Review the exact slice-B diff for the dedup/results cache, the Command arm, cache key construction, any shared state initialized at connect or first-command time, channel moves into the spawn, and guards around the Ack send.
+- Highest-yield hypotheses to check first:
+  - Dedup key is derived from command payload/action/verb instead of the unique request ID (so repeating "hostname" or any same-shape command hits as duplicate).
+  - Cache is never pruned / a global "seen" flag or inverted contains check.
+  - A response channel, JoinHandle, or oneshot is consumed on the first command and not recreated.
+  - The match/if that sends Ack is after a check that becomes permanently true after first use.
+- Build a minimal local repro (tokio-tungstenite server loop + the 0.6.63 agent binary or source). Send: Ping, Command (any), Ping, Command (same), Command (different). Observe whether exactly one Command ever gets Acked per process lifetime.
+- Add (or review) unit tests for handle_server_message covering first command, second distinct command, duplicate-ID retry, and reconnect-within-process.
+- In the next build (even the debug one), make received Text frames, pre/post-parse kind, cache hit/miss, and Ack decision noisy at debug level.
+
+c) Keep or roll back the SNAT rule?
+Roll it back. It was a diagnostic, produced no behavior change, and (per the reply-to caveat) probably never exercised the relevant return path anyway. Clean up the pfSense rule, confirm `pfctl -sn`, and move on. No value in keeping it.
+
+d) Anything else:
+- Treat the command ingestion path (especially the new durability pieces) as the sole focus until the one-command-then-stuck pattern is reproduced and fixed.
+- When you have a candidate fix, deploy it via the update path if possible (the restart-on-update gives a clean process) and immediately test multiple commands + reconnects on a lab or one of the recovered hosts.
+- Add defensive logging/metrics in the agent for command receive/Ack/dedup/execute counts so the next time this class of bug appears it is visible without packet captures.
+
+**4. Any remaining low-risk test before committing to "fix the agent"?**
+
+Yes, a few cheap ones that add signal without touching the production broken agents or the SNAT state:
+
+- On any lab/unaffected/can-restart-safely agent (or a local build), do a clean process restart then issue two different commands in quick succession (hostname then whoami/ipconfig or equivalent). If only the very first command after process start ever succeeds, this is near-conclusive for the cache/dispatch state bug and justifies going straight to the code fix.
+- Compare server-side logs and the exact JSON for the 22:23 successful command vs later failing ones (same ID format? same structure? any difference in how the command envelope was serialized?).
+- If you can get a short capture on a test machine right after restart + first command, watch the agent->server direction for whether a CommandAck Text frame is actually emitted (distinguishes "never sent by agent" vs "sent but lost").
+- Quick server-side sanity: confirm the command retry/delivery logic is not itself suppressing re-sends for an ID it thinks it already saw.
+
+If the "two distinct commands after restart" test shows the strict one-success-per-process-lifetime pattern, stop testing the network side and fix the agent. The data already points there with high confidence.

tmp/rmm-diag-dossier.md

@@ -0,0 +1,181 @@
+# GuruRMM — agent command delivery failure: diagnostic dossier (neutral)
+
+You are one of two independent AI models asked to help diagnose a distributed-systems
+issue. Please reason from the raw facts below. Do NOT assume any particular root cause is
+already established — several plausible mechanisms exist and we want them all on the table.
+
+At the end, give:
+1. Your ranked list of candidate root-cause hypotheses (with the reasoning/evidence for and against each).
+2. The SINGLE most diagnostic test or piece of data you want next (something concrete we can run),
+   and what each possible outcome would tell you.
+
+Be skeptical and specific. Call out where the evidence is contradictory or where a stated
+"fact" might actually be an unverified inference.
+
+---
+
+## 1. System architecture (factual)
+
+- **GuruRMM**: a remote monitoring/management system. A central server manages ~200 agents.
+- **Agent**: Rust binary (current fleet version 0.6.63). Runs as a service on Windows/macOS/Linux
+  endpoints. Connects OUTBOUND to the server over a WebSocket (WSS).
+- **Ingress path** for the agent WebSocket:
+  `agent  ->  (endpoint's LAN/NAT/router)  ->  Internet  ->  Cloudflare edge (rmm.azcomputerguru.com, 104.26.x / 2606:4700)  ->  origin host 172.16.3.30  ->  nginx (TLS :443)  ->  proxy_pass http://127.0.0.1:3001 (the Rust server)`
+  - nginx `/ws` location config: `proxy_read_timeout 86400;` plus the standard `Upgrade`/`Connection` headers for WebSocket.
+  - The origin server (172.16.3.30) has NO direct TCP connection from any endpoint's WAN IP; all agent
+    sockets terminate at nginx, which presents them to the Rust server as loopback (127.0.0.1) connections.
+- **Server**: Rust (axum + tokio + tokio-tungstenite). Listens 0.0.0.0:3001. One async task per agent connection.
+
+## 2. The agent<->server protocol (factual, from source)
+
+Server -> agent messages (all JSON over WS Text frames) include: AuthAck, ConfigUpdate, Command,
+Update, CancelCommand, plus WS-level Ping frames.
+
+Agent -> server messages (JSON over WS Text frames) include: Auth, Heartbeat, Metrics, CommandResult,
+CommandAck, UpdateResult, plus WS-level Pong frames (in response to server Ping).
+
+Connection lifecycle:
+- Agent connects, sends Auth. Server replies AuthAck, then sends a ConfigUpdate (policy; a multi-hundred-byte
+  to few-KB JSON Text frame), then re-offers any pending commands.
+- Server runs a per-connection send task that: (a) forwards queued server->agent messages to the socket,
+  and (b) sends a WS **Ping** frame every **30 seconds** (NAT keepalive).
+- Agent read loop: on EVERY inbound frame it resets a deadline. If it receives **no inbound frame for 90s**,
+  it declares the connection dead and reconnects (jittered). It responds to server WS Ping with WS Pong.
+- Agent sends a Heartbeat (Text) and Metrics on their own cadences (Heartbeat roughly every ~2 min based on
+  server logs; Metrics more often). The server updates the agent's `last_seen` on inbound messages.
+
+Command delivery + durability (shipped earlier today, "comms-durability Phase 1"):
+- Command lifecycle in DB: `pending` -> `running` (on dispatch; `delivery_attempts` increments each dispatch)
+  -> server stamps `acked_at` when the agent sends **CommandAck** -> `completed`/`failed` on result.
+- Agent (0.6.63) sends **CommandAck on RECEIPT of a command, BEFORE executing it** (first thing in its
+  command handler), then dedups (won't re-run a command id it's already running/done).
+- Server reaper runs every 60s: (1) returns un-acked `running` commands older than 60s to `pending`
+  (re-delivery, never fail) for ACK-capable agents; (2) re-offers pending commands on every agent Heartbeat
+  AND on (re)connect; (3) fails a command only after 10 delivery attempts (undeliverable) or a real post-ACK
+  execution timeout.
+- `send_to(agent)` = look up the agent's sender channel in an in-memory map and `tx.send(msg)`; returns true
+  if the channel accepted the message.
+
+## 3. The symptom
+
+For a SUBSET of agents (confirmed: PST-SERVER, PST-SERVER2 — same physical site, behind a UniFi UDR
+"Cloudflare-Ultra"-class gateway; and GTS-PEDRO-H — a DIFFERENT site), server->agent COMMANDS are not
+executed: the command sits in `running`, `acked_at` never gets set, `delivery_attempts` climbs, and it
+eventually fails. Meanwhile the SAME agents keep heartbeating and the server keeps showing them online.
+
+Most of the fleet is UNAFFECTED: in the last 6 hours 397 commands completed across 40 distinct agents,
+most recent completion timestamped AFTER the symptom window.
+
+## 4. Per-agent observations (raw)
+
+### PST-SERVER (agent id 87293069..., v0.6.63)
+- `agent_events` shows it went `offline` at 00:22:00 then `online` at 00:22:03, and then NO further
+  online/offline events for the next 30+ minutes (a single continuous connection from the server's view).
+- Throughout that window the server's `last_seen` for it stayed current to the second (i.e. the server
+  kept receiving inbound messages — heartbeats/metrics — from the agent).
+- Commands dispatched in that window:
+  - 22:23:40 (before the 00:22 reconnect, on a prior connection): `hostname; whoami` -> **completed**,
+    `acked=true`, delivery_attempts=1, ack latency 16 ms. (This was minutes after the agent updated to 0.6.63.)
+  - 00:29:17: `powershell.exe -NonInteractive -EncodedCommand <~4 KB base64>` -> never acked, attempts climbed
+    to 10, failed.
+  - 00:38, 00:47 etc.: same large command shape -> never acked.
+  - A **tiny** `hostname` command (~80-byte Text frame) dispatched in the same period -> also never acked.
+- Server logs during the stuck window, repeating every cycle:
+  `Reaper: re-delivered 1 un-acked command(s)` then `Re-offering 1 pending command(s) to agent 87293069...`.
+  The re-offer's `send_to` returns success (no "failed to enqueue" warning).
+
+### PST-SERVER2 (agent id 5d2d7ba0..., v0.6.63, same site)
+- Has NEVER successfully acked a command (no `acked_at` ever recorded). Commands fail as "undeliverable".
+- `last_seen` is current (server receives its inbound heartbeats/metrics).
+
+### GTS-PEDRO-H (agent id 2f1499f8..., v0.6.63, different site)
+- 2026-06-12 01:43: `hostname` (tiny) -> **failed**, acked=false, attempts=1 (failed at the 60s command timeout).
+- Last prior successful command: 2026-06-06 (6 days earlier; those completed but with acked=false — note those
+  predate the CommandAck feature so acked=false then is expected). No commands attempted between 06-06 and 06-12.
+
+### Fleet baseline
+- 397 commands `completed` in the last 6h across 40 distinct agents; most recent completion 01:44:43Z.
+
+## 5. Tests performed and their RAW results
+
+T1. Repeatedly dispatched the large (~4 KB) recon command to PST-SERVER. Result: never acked; attempts 1->10;
+    failed with stderr "Command undeliverable — agent unreachable" or "Command timed out".
+T2. Dispatched a tiny `hostname` (~80 B Text frame) to PST-SERVER while it was in the stuck state.
+    Result: never acked; attempts climbed; did not execute.
+T3. Cancelled ALL pending/running commands for PST-SERVER (to clear any queue/head-of-line buildup), waited 35s,
+    then dispatched a single tiny `hostname`. Result: still never acked.
+T4. Verified PST-SERVER's `agent_events`: exactly one online (00:22:03) over 30+ minutes — i.e. the agent did
+    NOT hit its own 90s "no inbound" reconnect during the stuck window (implication: it was receiving SOME
+    inbound frames, at least the 30s server Pings, within every 90s window).
+T5. Verified the server keeps receiving the agent's inbound: `last_seen` current; the per-Heartbeat
+    "Re-offering pending command" log fires, which is triggered by an inbound Heartbeat from the agent.
+T6. Inspected ingress: nginx fronts the server; Cloudflare fronts nginx; `proxy_read_timeout 86400` on `/ws`.
+    No direct WAN socket from the endpoint on the origin (all agent sockets are loopback-from-nginx there).
+T7. Deployed a SERVER change: a background task that force-closes (evicts) a connection once one of its
+    un-acked commands has been re-offered >= 3 times, to force the agent to reconnect onto a new connection.
+    Verified from logs that eviction fired and the agent reconnected ~12s later:
+      `Reaper: evicted non-delivering connection for agent 87293069 (forcing reconnect)`
+      `... connection evicted ... closing to force a fresh reconnect`
+      `ws: interrupted 1 orphaned commands for agent 87293069`  (reconnect handshake)
+      `Sent ConfigUpdate to agent 87293069`                      (reconnect handshake)
+      `Agent 87293069 reconnected during grace period`
+    So eviction -> reconnect works mechanically.
+T8. On the FRESH post-reconnect connection, dispatched a tiny `hostname`. Result: STILL never acked
+    (attempts climbed 1->2, no ack). (NOTE: "Sent ConfigUpdate" in T7 only means the server WROTE the frame to
+    its socket; we have NO independent confirmation the agent RECEIVED the ConfigUpdate.)
+T9. Control test: dispatched a tiny `hostname` to GTS-PEDRO-H (non-PST, 0.6.63). Result: failed, never acked.
+T10. Fleet check: 40 distinct agents have completed commands in the last 6h; delivery is NOT universally broken.
+
+## 6. Relevant code excerpts (factual)
+
+Server per-connection send task (one per agent):
+```rust
+let mut send_task = tokio::spawn(async move {
+    let mut ping_interval = tokio::time::interval(Duration::from_secs(30));
+    loop {
+        tokio::select! {
+            msg_opt = rx.recv() => { match msg_opt {
+                Some(msg) => { if sender.send(Message::Text(json)).await.is_err() { break; } }
+                None => break, } }
+            _ = ping_interval.tick() => { if sender.send(Message::Ping(vec![])).await.is_err() { break; } }
+        }
+    }
+});
+```
+
+Server receive loop (same connection): handles agent Text via handler; on `Message::Ping` replies an Ack;
+`Message::Pong` and other frames fall through to `_ => {}` (ignored).
+
+Agent read loop:
+```rust
+Some(msg_result) = read.next() => {
+    last_incoming = Instant::now();                  // reset on EVERY inbound frame
+    match msg_result {
+        Ok(Message::Text(text)) => { handle_server_message(&text, ...).await }  // Command handled here -> sends CommandAck first
+        Ok(Message::Ping(data)) => { write.send(Message::Pong(data)).await }
+        Ok(Message::Pong(_))    => { /* debug */ }
+        Ok(Message::Close(_))   => break Ok(()),
+        Err(e)                  => break Err(e),
+    }
+}
+_ = sleep_until(last_incoming + Duration::from_secs(90)) => {   // no inbound for 90s
+    break Err(anyhow!("Connection timeout: no server activity"));
+}
+```
+
+Agent command handler (executor), first action on receiving a Command:
+```rust
+// runs as soon as a ServerMessage::Command is parsed:
+tx.send(AgentMessage::CommandAck { command_id }).await   // ACK on receipt, BEFORE executing
+// then dedup (cached/in-flight) and execute
+```
+
+## 7. What we explicitly do NOT yet know
+- Whether the agent at PST-SERVER is actually RECEIVING the server's Command Text frames (we only know it is
+  NOT sending CommandAck, and is NOT executing them).
+- Whether the agent is receiving the server's 30s Ping frames (inferred ONLY from "it didn't reconnect in 90s").
+- Whether the server's ConfigUpdate frame on reconnect actually reached the agent.
+- We have NO packet capture and NO agent-side logs (we cannot currently reach these endpoints to pull logs,
+  because the only remote-control channel IS the thing that's broken).
+
+Please diagnose.

tmp/rmm-diag-round2.md

@@ -0,0 +1,84 @@
+# GuruRMM command-delivery diagnosis — ROUND 2 (important topology correction)
+
+You analyzed this in round 1 (your prior answer is included at the bottom). A FACTUAL CORRECTION
+to the architecture has come to light that invalidates part of the round-1 premise. Please
+re-evaluate from scratch where needed. Be willing to discard your prior top hypothesis.
+
+## CRITICAL CORRECTION to the ingress topology
+
+In round 1 we told you: "agent -> Cloudflare -> nginx -> Rust server". **That was wrong for the
+AGENT path.** Verified facts now:
+
+- The agent's hard-coded WebSocket URL is **`wss://rmm-api.azcomputerguru.com/ws`** (from agent source:
+  `DEFAULT_SERVER_URL`, and the agent config default). The installer and enrollment also use
+  `rmm-api.azcomputerguru.com`.
+- DNS / Cloudflare proxy status for the two hostnames (both A-record to the same public IP 72.194.62.10):
+  - `rmm.azcomputerguru.com`     -> **proxied = true**  (orange cloud; goes THROUGH Cloudflare)  — this is the human DASHBOARD.
+  - `rmm-api.azcomputerguru.com` -> **proxied = false** (grey cloud; **DNS-only, BYPASSES Cloudflare**) — this is what the AGENTS use.
+- Therefore **Cloudflare is NOT in the agent's path at all.** All round-1 hypotheses about Cloudflare
+  WAF / CF WebSocket buffering / CF frame handling are moot for agent command delivery. (Cloudflare only
+  fronts the dashboard.)
+
+## The ACTUAL agent path (verified)
+
+```
+agent (endpoint, e.g. PST-SERVER behind a UniFi UDR/"Cloudflare-Ultra"-class gateway)
+  -> endpoint LAN/NAT
+  -> public internet
+  -> 72.194.62.10  (public IP; this is the NPM box)
+  -> NPM = "Nginx Proxy Manager" on host 172.16.3.20  (terminates TLS; one nginx layer)
+       NPM proxy-host settings for rmm-api: forward http://172.16.3.30:80, websockets=ON,
+       http2_support=ON, block_exploits=ON (NPM "Block Common Exploits"), caching=OFF, advanced_config=EMPTY
+  -> http://172.16.3.30:80   (the ORIGIN nginx; a SECOND nginx layer)  -- PLAINTEXT HTTP over the LAN here
+  -> proxy_pass http://127.0.0.1:3001   (the Rust server)
+```
+
+So there are **TWO nginx proxy layers** in series (NPM on .20, then origin nginx on .30), no CDN.
+
+Origin nginx `/ws` block (verbatim):
+```
+location /ws {
+    proxy_pass http://127.0.0.1:3001;
+    proxy_http_version 1.1;
+    proxy_set_header Upgrade $http_upgrade;
+    proxy_set_header Connection "upgrade";
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_read_timeout 86400;
+}
+```
+(No explicit `proxy_buffering off;`, no `proxy_send_timeout`. NPM's generated config for the proxy host is
+the standard NPM template with websockets enabled + block_exploits; we have not yet dumped its exact directives.)
+
+## What this changes / things to reconsider
+
+- The "asymmetry" symptom is unchanged and still the core puzzle: for affected agents the server keeps
+  receiving inbound (heartbeats, last_seen fresh), the agent does NOT trip its 90s no-inbound reconnect
+  (so it appears to be receiving at least the server's 30s WS Ping frames), yet server->agent COMMAND (Text)
+  frames are never acked/executed — both ~4 KB commands AND ~80 B `hostname` commands. One command DID
+  succeed earlier (right after the agent updated/reconnected). A server-forced eviction + fresh reconnect did
+  NOT restore command delivery (a tiny command on the fresh connection still never acked) — but note we have
+  NO confirmation the agent received anything server->agent on that fresh connection either.
+- Affected: PST-SERVER + PST-SERVER2 (same site/UDR), and GTS-PEDRO-H (different site). ~40 other agents
+  deliver commands fine through this SAME NPM+nginx path.
+- Because rmm-api is grey-cloud, the agent's TLS connection goes straight from the endpoint's gateway to the
+  public IP — there is no CDN normalizing MTU/keepalive/buffering on the path; the endpoint's own gateway
+  (e.g. PST's UDR) is directly in the path.
+
+## A note on one earlier "test"
+
+We tried to `tcpdump` the origin loopback (nginx->Rust on 127.0.0.1:3001) during a marked command dispatch and
+saw 0 packets — but that capture almost certainly FAILED (the backgrounded tcpdump was killed when the SSH
+session closed), so treat it as NO DATA, not as evidence that the loopback is silent.
+
+## Please now provide (round 2)
+
+1. Your REVISED ranked hypotheses given Cloudflare is out of the agent path and there are two nginx layers +
+   the endpoint gateway directly in the path. Explicitly say which round-1 hypotheses you drop or demote.
+2. The single most diagnostic next test (we can run packet captures on the .30 host on the plaintext .20->.30:80
+   leg and on the 127.0.0.1:3001 loopback; we can read NPM's generated nginx config on .20; we can query the
+   UDR; we currently CANNOT get shell/logs on the affected endpoints). For each plausible outcome, what it tells you.
+
+---
+## YOUR ROUND-1 ANSWER (for reference)
+__ROUND1__

tmp/rmm-diag-round3.md

@@ -0,0 +1,54 @@
+# GuruRMM command-delivery diagnosis — ROUND 3 (specific proposal to evaluate)
+
+You are part of a two-model quorum diagnosing why a subset of RMM agents stop receiving
+server->agent WebSocket COMMAND (Text) frames while everything else about their connection
+looks healthy. Rounds 1-2 established (summarized):
+
+- Topology (agent path, NO Cloudflare — agents use grey-cloud `rmm-api`):
+  `agent -> endpoint gateway (e.g. UniFi UDR) -> public internet -> public IP 72.194.62.10
+   -> NPM (Nginx Proxy Manager, host .20, TERMINATES TLS) -> http://172.16.3.30:80 (origin nginx, 2nd layer)
+   -> proxy_pass 127.0.0.1:3001 (Rust server)`. Two nginx layers in series.
+- Symptom: affected agents (PST-SERVER, PST-SERVER2 same site/UDR; GTS-PEDRO-H different site; all v0.6.63)
+  keep heartbeating (agent->server fine, last_seen fresh), do NOT trip their 90s no-inbound reconnect
+  (so they seem to receive at least the server's 30s WS Ping), but never ACK/execute commands (tiny ~80B AND
+  large ~4KB). ~40 other agents on the SAME NPM+nginx path receive commands fine. A forced server-side
+  eviction + fresh reconnect did NOT restore delivery.
+- Current top hypotheses (both models, round 2): (H1) one of the two nginx layers mishandles/buffers
+  server->agent Text frames after the WS upgrade (origin `/ws` lacks an explicit `proxy_buffering off;`);
+  (H2) agent-side 0.6.63 receive/command path defect that survives reconnect; (H3) endpoint gateway
+  (UDR) selectively interferes with downstream Text frames on the un-CDN'd direct path. Cloudflare and MTU
+  were dropped.
+- We have NOT yet localized where a Text frame dies (packet capture tooling was just installed; not run yet).
+
+## The proposal to evaluate (from the operator)
+
+**Remove NPM (Nginx Proxy Manager) from the equation entirely — NAT the public traffic straight to .30.**
+
+Relevant constraints / facts:
+- NPM (.20) is currently the PUBLIC TLS terminator for BOTH `rmm.azcomputerguru.com` (dashboard, Cloudflare
+  orange-cloud -> NPM) AND `rmm-api.azcomputerguru.com` (agents, grey-cloud DNS-only -> straight to the public
+  IP -> NPM). NPM holds the TLS certs.
+- The origin nginx on .30 currently listens on :80 PLAINTEXT only (no :443, no TLS cert installed there today).
+- Agents connect with `wss://rmm-api.azcomputerguru.com/ws` — TLS is mandatory for them.
+- So "NAT straight to .30" implies the edge firewall/router DNATs public :443 to .30, and .30's nginx must be
+  reconfigured to terminate TLS on :443 (install/move the cert) and proxy `/ws` -> 127.0.0.1:3001. Net effect:
+  ONE nginx layer instead of two; NPM out of the agent path.
+- ~200 agents total. The dashboard path (`rmm`, Cloudflare) would also need a decision (keep it on NPM, or
+  also move it).
+
+## Please answer (each model, concise but rigorous)
+
+1. As a DIAGNOSTIC bisection: if we remove NPM and the affected agents (PST-SERVER etc.) START receiving
+   commands, what does that prove? If they STILL don't, what does that prove? Is this a clean bisection of the
+   current hypothesis set (H1 nginx-layer vs H2 agent vs H3 gateway), or are there confounds?
+2. As a permanent FIX/SIMPLIFICATION: pros/cons of collapsing to a single nginx layer on .30. What could break
+   or regress (TLS/cert handling, the Cloudflare-fronted dashboard, HTTP/2, the grey-cloud direct exposure of
+   .30 to the internet, security surface, cert renewal, etc.)?
+3. Is there a LOWER-RISK / FASTER way to get the SAME diagnostic signal WITHOUT re-NATing production — e.g. a
+   parallel test listener, a single test agent pointed at a bypass hostname/port, an `/etc/hosts` override on
+   one endpoint, or adding `proxy_buffering off;`/WS hardening to one layer first? If so, specify it.
+4. Net recommendation: do the NAT-straight-to-.30 change now, or do a specific cheaper test first? Give a
+   concrete ordered action list.
+
+Be specific and call out anything in the operator's proposal that is risky or that wouldn't actually isolate
+the variable they think it does.

tmp/rmm-diag-round4.md

@@ -0,0 +1,54 @@
+# GuruRMM command-delivery diagnosis — ROUND 4 (new packet-capture evidence)
+
+Quorum of two models, continuing. New hard evidence has localized the failure further. Please
+update your hypothesis ranking and pick the single best next action.
+
+## Recap (established)
+- Agent path (NO Cloudflare; agents use grey-cloud rmm-api): `agent -> endpoint gateway (UniFi UDR) ->
+  internet -> public IP 72.194.62.10 -> edge firewall DNAT -> NPM (Nginx Proxy Manager, host .20, Docker
+  on an Unraid box; TERMINATES TLS) -> http://172.16.3.30:80 (origin nginx) -> 127.0.0.1:3001 (Rust server)`.
+- Symptom: affected agents (PST-SERVER, PST-SERVER2 same site/UDR; GTS-PEDRO-H different site; all v0.6.63)
+  heartbeat fine (agent->server OK), don't trip their 90s no-inbound reconnect, but never ACK/execute
+  server->agent commands (tiny 80B AND 4KB). ~40 other agents on the SAME path get commands fine.
+
+## NEW EVIDENCE — packet capture on the origin host (.30), both legs simultaneously
+We dispatched 5 uniquely-marked tiny `hostname-FINALMARKn-www` commands to the stuck PST-SERVER while
+capturing on .30 (loopback + LAN). Findings (payloads are PLAINTEXT — WS permessage-deflate is NOT in use,
+so frames are directly readable; agent->server frames are masked as expected):
+
+- On the **loopback :3001** leg (Rust server <-> origin nginx): the command Text frame is present, e.g.:
+  `localhost.3001 > localhost.54036  {"type":"command","payload":{"id":"c109e270...","command":"hostname-FINALMARK1-www",...}}`
+  => **The Rust server DID emit the WS Text frame.**
+- On the **.30 -> .20:80** leg (origin nginx -> NPM): the SAME command frame is present, e.g.:
+  `gururmm.http > 172.16.3.20.45614  {"type":"command",...FINALMARK1...}`
+  => **The origin nginx DID forward the frame out toward NPM (.20).**
+- So the frame provably traverses the Rust server AND the origin nginx and is sent to NPM. It still never
+  reaches the agent (no ACK, no execution).
+- (Minor: the marker appeared 13x on loopback vs 8x on the LAN leg over the window — possibly retransmit/
+  re-offer timing; not yet interpreted. We did NOT capture on NPM (.20) or on the agent.)
+
+## What this rules in/out
+- Refuted: "Rust send is lying" and "origin nginx (.30) swallows/buffers the frame" — the frame leaves both.
+- The death is now localized to: **NPM (.20) itself, OR the NPM->agent leg (public internet / the endpoint's
+  UDR gateway), OR the agent.**
+
+## Constraints on next tests
+- We CAN now capture on the NPM box (.20, Unraid host; we have root). The NPM->agent leg is TLS (encrypted),
+  so on .20 we could see whether NPM emits TLS application-data toward the agent's WAN IP (98.190.129.150 for
+  PST) shortly after receiving each command, and whether those packets are ACK'd or retransmitted (a black-hole
+  signature) — but we cannot read the frame contents on that leg.
+- We have a CATCH-22 for any "point the agent at a bypass" test: the only way to reconfigure an affected
+  agent's server URL is to send it a command/config — which is the very channel that's broken. There is no
+  known second, WORKING agent at the PST site. (On-site access would be required to re-point an agent there.)
+- We can freely add config to NPM (via its API/advanced-config) and to the origin nginx, and reload.
+- ~40 other agents work fine through this identical NPM. The affected set is a subset.
+
+## Please answer (each model, concise)
+1. Updated ranked hypotheses given the frame reaches NPM but not the agent. Where do you now put: NPM
+   buffering/relay defect; NPM->WAN/UDR black-hole (H3); agent-side defect (H2)? What does "subset of agents
+   on the same NPM" imply about NPM-internal vs per-connection-WAN causes?
+2. The single most decisive next test we can actually run (given we can capture on .20 but canNOT easily
+   reconfigure the broken agents). For each outcome, what it tells us. Specifically: is a .20 capture of the
+   NPM->agent leg (encrypted; flow/retransmit analysis only) decisive enough, or is there a better move?
+3. Does this new evidence change the verdict on the operator's "remove NPM / NAT straight to .30" idea — is it
+   now more justified, or is there still a cheaper test that isolates NPM specifically?

tmp/rmm-diag-round5.md

@@ -0,0 +1,56 @@
+# GuruRMM command-delivery diagnosis — ROUND 5
+
+Quorum of two models, continuing. New config + population evidence. Re-rank and propose the deepest
+useful diagnostic plan; we have several more rounds available, so think hard about what would actually
+discriminate the remaining hypotheses.
+
+## Where we are (established by packet capture)
+- Agent path (NO Cloudflare): `agent -> site gateway -> internet -> public IP -> NPM (Nginx Proxy Manager,
+  Docker bridge container on Unraid host .20, TERMINATES TLS) -> http://.30:80 (origin nginx) -> 127.0.0.1:3001 (Rust)`.
+- A marked command Text frame provably (a) leaves the Rust server (loopback :3001 capture) and (b) is forwarded
+  by the origin nginx out toward NPM (.30->.20:80 capture). It never reaches the agent (no CommandAck, no exec).
+- So the failure is in the NPM(.20)->agent segment. Everything upstream is exonerated.
+- A capture on the NPM box of the NPM->agent leg is in progress (results not yet in).
+
+## NEW: the affected population vs the working majority (this is the key analytical puzzle)
+Affected agents and their PUBLIC IPs (from agent-reported metrics) — note THREE DIFFERENT ISPs/sites:
+- PST-SERVER     98.190.129.150   (site A; UniFi UDR gateway)
+- PST-SERVER2    64.139.88.249    (site B, different physical site/ISP from PST-SERVER; also a UniFi gateway)
+- GTS-PEDRO-H    68.230.27.220    (site C, different ISP again)
+All three are v0.6.63. ~40 OTHER agents on the SAME NPM + SAME 0.6.63 receive commands fine.
+
+So the root cause must be something COMMON to these three and ABSENT from the 40 working ones. It is NOT:
+- a single site's gateway (three different gateways/ISPs),
+- the NPM proxy globally (40 work through it),
+- the agent version globally (40 work on 0.6.63).
+What could the shared factor be? (e.g., all three are WINDOWS SERVERS rather than workstations? long-lived/old
+connections? a particular enrollment/update cohort? NIC/TCP-offload on server hardware? a specific policy/config
+pushed to them? connection idle pattern? Please reason about what class of shared attribute fits.)
+
+## NEW: NPM internal config (dumped from the container)
+- NPM proxy host for rmm-api/rmm: `http2 on;`, WS headers present (`proxy_set_header Upgrade $http_upgrade;
+  proxy_set_header Connection $http_connection; proxy_http_version 1.1;`), `include block-exploits.conf;`,
+  forwards `http://172.16.3.30:80`. NPM's `proxy.conf` has **no `proxy_buffering off;` and no
+  `proxy_read_timeout` override** -> nginx DEFAULTS apply (proxy_buffering on; proxy_read_timeout 60s).
+- NPM error log: many `"an upstream response is buffered to a temporary file while reading upstream"` warnings,
+  but ALL are for NON-WebSocket HTTP requests (/downloads/ the agent .exe, /api/ GETs, /assets/*). NONE for the
+  WS connections and NONE mentioning the affected agents' IPs.
+- NPM container is bridge-networked; inside the container, client source IPs are NAT-masked by Docker, which
+  complicates per-connection correlation there (we are capturing on the host instead).
+
+## Questions for round 5
+1. Given three DIFFERENT ISPs/sites all show the identical "control frames + heartbeats fine, server->agent
+   Text frames never delivered" signature, re-rank: NPM-relay/state vs per-site-gateway (H3) vs agent-side (H2).
+   The three-different-paths fact seems to argue strongly against independent per-site network black-holes.
+   What shared attribute best explains "these 3 but not the other 40"? Be concrete and falsifiable.
+2. Does `http2 on` at NPM combined with default `proxy_buffering on` and a WS upgrade create any known failure
+   mode for server->client data frames on long-lived connections (vs client->server which works)? Is there a
+   known nginx/NPM bug where the *upgraded* connection's downstream (server->client) frames can stall under
+   specific conditions while pings/heartbeats pass?
+3. Propose the most discriminating SEQUENCE of tests for the next 2-3 rounds (we can: capture on .20 host;
+   read NPM/agent TCP socket stats via ss on the host; restart the NPM container as a one-shot probe [costs a
+   brief reconnect of all ~200 agents]; add/remove NPM config like `proxy_buffering off` or `http2 off` and
+   reload; query the affected agents' DB rows for any shared policy/enrollment attribute; we CANNOT push config
+   to the broken agents nor get on-site easily). Order them by (information gained / risk).
+4. Specifically: is restarting the NPM container a worthwhile one-shot probe now, and exactly what would
+   "stuck agents recover after NPM restart" vs "they do not" each prove?

tmp/rmm-diag-round6.md

@@ -0,0 +1,67 @@
+# GuruRMM command-delivery diagnosis — ROUND 6 (decisive capture + reconcile a contradiction)
+
+Quorum continuing. A clean packet capture has produced a result that seems to CONTRADICT the leading
+network/proxy hypotheses. Please reconcile it and pick the decisive next test.
+
+## Path recap (no Cloudflare for agents)
+agent -> site gateway -> internet -> pfSense (multi-WAN, public IP blocks 72.194.62.x AND 70.175.28.x;
+DNAT wan:443 -> NPM 172.16.3.20:18443) -> NPM (Nginx Proxy Manager, terminates TLS) -> origin nginx .30:80
+-> Rust server :3001. Affected: PST-SERVER (98.190.129.150), PST-SERVER2 (64.139.88.249),
+GTS-PEDRO-H (68.230.27.220) — 3 different ISPs/sites, all v0.6.63. ~40 other 0.6.63 agents work fine.
+
+## DECISIVE NEW CAPTURE (on the NPM/Jupiter host, of the NPM<->PST-agent leg, during 3 marked commands)
+- Dispatched 3 tiny commands (CLEANA/B/C) to PST-SERVER at known times.
+- NPM->agent direction showed exactly THREE 214-byte TLS frames (= the 3 commands; a hostname command is a
+  193-byte WS Text frame + ~21B TLS overhead) interleaved with 31-byte frames (the server's WS Pings).
+- The TCP sequence numbers advanced cleanly across the 214B frames with NO retransmissions (the apparent 3x
+  duplication is just `tcpdump -i any` capturing each packet on veth+bridge+host).
+- Interpretation: NPM DID forward all three command frames to the agent, and the agent's TCP ACKed them
+  (no retransmit). i.e. the command BYTES reached the agent's kernel/TCP stack.
+- Yet: the agent never sent a CommandAck and never executed any of them (server-side acked_at stays null).
+- Earlier captures already proved the frame leaves Rust and leaves origin nginx. And the agent on this same
+  connection continues to answer the server's pings (it does not hit its 90s no-inbound reconnect) and sends
+  heartbeats/metrics that DO reach the server (last_seen stays fresh; ~1346B agent->server frames were seen).
+
+## What this seems to rule out
+- NPM swallowing the frame (it forwarded all 3).
+- WAN black-hole / wrong-source SNAT on the server->agent direction (a wrong source IP would cause the agent
+  NOT to ACK -> retransmissions; we saw clean ACKs, no retransmits).
+- The frame never reaching the agent.
+
+## pfSense facts (operator suspected an SNAT/return-path issue)
+- Outbound NAT mode = hybrid; multi-WAN (WAN + FIBER).
+- DNAT: wan:443 -> 172.16.3.20:18443 (NPM). (There is also a near-duplicate "Emby on Fiber" wan:443->.20:18443.)
+- Manual outbound-NAT (SNAT) exists for src 172.16.3.10 -> 72.194.62.5, but there is NO explicit SNAT pinning
+  NPM (172.16.3.20) -> 72.194.62.10; NPM replies rely on pf state/reply-to + automatic outbound NAT.
+- BUT the capture shows the agent ACKing the server->agent command frames, which argues the reply path is
+  currently delivering with an acceptable source IP. So the SNAT config gap, while real, does not appear to be
+  dropping these particular server->agent frames.
+
+## Agent self-logs
+- The agent uploads logs to the server (agent->server works), but the last uploaded batch for PST-SERVER is
+  from 22:21 (the moment it updated to 0.6.63 — at which point it successfully applied a ConfigUpdate, sent
+  inventory, and at 22:23 ONE command succeeded+acked). No fresh logs since (log upload is server-triggered =
+  broken channel). So we cannot see the agent's current-state logs remotely.
+
+## The contradiction to resolve
+The command bytes provably reach the agent's TCP and are ACKed, the agent processes the server's PING control
+frames on the SAME connection (so its read loop is alive), yet it does not parse/ack/execute the command TEXT
+frames. It worked once at 22:23 right after the 0.6.63 update, then stopped.
+
+## Questions (each model, concise)
+1. Given the command bytes are TCP-ACKed by the agent but never produce a CommandAck, and the same connection
+   still processes pings: what agent-side mechanisms can make a WS client ACK TCP + handle Ping control frames
+   but NOT deliver Text data frames to the application? (e.g., TLS record vs WS-frame reassembly state; a stuck
+   read between the TLS layer and the WS dispatch; a per-connection WS read buffer that desyncs after a certain
+   point; recv-buffer filling because the app stopped reading while the kernel keeps ACKing; a panic/dropped
+   task in the command dispatch path; etc.) Rank them.
+2. Is "kernel ACKs while the application stopped reading" plausible here, and how would the pings still get
+   answered if the app stopped reading? (Reconcile carefully.)
+3. Could anything OTHER than the agent still explain TCP-ACKed-but-not-processed (e.g., a middlebox at the
+   agent's own site doing TCP proxying/termination that ACKs then mangles; TLS-inspection at the agent's UDR;
+   GRO/LRO/TSO offload on the server-grade NICs corrupting reassembly only for these hosts)? Weigh these.
+4. The single most decisive next test. We CAN: capture at any point in OUR infra (pfSense/NPM/origin), read the
+   server DB, read the agent's LAST uploaded logs, change pfSense/NPM/server config, force the agent to
+   reconnect (server-side eviction), restart NPM. We CANNOT push config to the broken agents or get on-site
+   quickly. What single action best discriminates "agent app" vs "agent-site middlebox/offload" vs "still NPM/pf"?
+5. Does the "worked at 22:23 right after update, never since" timing point anywhere specific?

tmp/rmm-diag-round7.md

@@ -0,0 +1,43 @@
+# GuruRMM command-delivery diagnosis — ROUND 7 (SNAT test result)
+
+Quorum continuing. The operator's SNAT/return-path hypothesis was tested directly. Result below — please
+give your final verdict and the recommended action plan.
+
+## Recap of the converging picture
+- agent -> site gateway -> internet -> pfSense (multi-WAN, IP-alias 72.194.62.10 on igc0/"wan") -> DNAT to
+  NPM (.20) -> origin nginx (.30) -> Rust :3001. Affected: PST-SERVER/PST-SERVER2/GTS-PEDRO-H (3 ISPs), v0.6.63.
+- Packet capture proved: command Text frame leaves Rust, leaves origin nginx, is forwarded by NPM to the agent,
+  and is TCP-ACKed by the agent with NO retransmits — i.e. the bytes reach the agent's kernel — yet the agent
+  never emits a CommandAck or executes, while still answering the server's pings on the same connection.
+- It worked for exactly ONE command at 22:23, immediately after the agent process restarted into 0.6.63
+  (this morning's "comms-durability slice B" added CommandAck-on-receipt + a dedup/results cache to the agent's
+  command path). A WS reconnect does NOT fix it (server restart + a forced eviction both gave fresh WS
+  connections, commands still failed). Only the agent PROCESS restart (the update) ever yielded a success.
+- Agent code note: the agent's command handler is inline in the read loop (handle_server_message().await),
+  sending CommandAck as its FIRST line before spawning the run task; a parse error at
+  serde_json::from_str::<ServerMessage>() is logged-and-ignored, so the read loop survives and keeps answering
+  pings. So "no CommandAck at all" implies the failure is at/just-after parse/dispatch, not a dead long-lived task.
+
+## THE SNAT TEST (operator's hypothesis, now tested directly)
+- Added a pfSense outbound-NAT (SNAT) rule pinning NPM source: `nat on igc0 inet from 172.16.3.20 to any ->
+  72.194.62.10`. Verified active in `pfctl -sn`. Fleet stayed healthy (149 agents online).
+- Killed PST-SERVER's pfSense states (forcing a brand-new TCP/WS connection under the new SNAT rule). PST
+  reconnected (last_seen fresh).
+- Dispatched a fresh `hostname` command on that new connection.
+- RESULT: still never ACKed (delivery_attempts climbing, no execution) — identical symptom.
+- Caveat we want you to weigh: pfSense replies to an INBOUND DNAT connection use reply-to/state, which
+  typically BYPASSES outbound NAT. So this SNAT rule may only affect NPM-INITIATED outbound, not the
+  server->agent reply path — i.e. the test may not have actually changed the reply source IP. However, the
+  earlier capture already showed the agent ACKing the server->agent frames (so the reply source was already
+  acceptable to the agent).
+
+## Questions (each model, concise)
+1. Does the SNAT test result (no change) — together with the capture showing commands reaching + being ACKed
+   by the agent — fully close the network/SNAT/proxy hypothesis class? Or is there a residual network angle the
+   SNAT test did NOT actually exercise (given reply-to bypasses outbound NAT)?
+2. Final verdict on the most probable root cause and its location, with confidence.
+3. Concrete recommended action plan, ordered: (a) immediate recovery of the 3 agents, (b) how to pin the exact
+   0.6.63 agent bug given we cannot reach the agents (can we infer it from the slice-B code shape: CommandAck +
+   dedup cache + per-command spawned task, inline in the read loop?), (c) whether to keep or roll back the SNAT
+   rule, (d) anything else.
+4. Is there any remaining LOW-RISK test that would add real information before we commit to "fix the agent"?

wiki/clients/peaceful-spirit.md

@@ -122,7 +122,7 @@ Massage therapy practice with at least two sites: Country Club (primary, all wor
 - **UDR Ultra reboot can silently drop the VPN port-forward (site-wide outage risk).** Confirmed 2026-06-04: the UDR Ultra (UCG-PST-CC) rebooted at 03:59 and came back without the UDP 500/4500 -> 192.168.0.2 port-forward, taking the entire site VPN offline with error 789 (IKE packets silently dropped at the edge). The `/data/on_boot.d/10-vpn-portforward.sh` persistence script was present but the UniFi OS 5.1.15 controller schema migration appears to have superseded it. **After any site-wide error 789, check the UDR port-forward in the UniFi controller FIRST** — IPsec auditing on the server (zero IKE events) is the confirmatory test. Long-term open items: (1) verify the re-added rule persists across a deliberate reboot (possible firmware bug or uncommitted rule), (2) add a DDNS hostname so the hardcoded 98.190.129.150 in client profiles is not a single point of failure for a Cox WAN-IP change.
 - **UCG iptables DNAT required — UniFi Traffic Rules are firewall-allow only, NOT DNAT.** Port-forward rules must be managed via the UniFi controller UI; `/data/on_boot.d/10-vpn-portforward.sh` is a legacy CLI fallback and may not persist reliably on UniFi OS 5.1.15+ (see above). Always verify iptables live after a reboot.
 - **UCG SSH unreachable from office WAN.** All remote UCG administration must go through GuruRMM (for PST-SERVER) or the UniFi cloud portal (for UCG itself). LAN SSH (192.168.0.10) requires keyboard-interactive auth — password auth via plink fails; use paramiko with kb_handler or interactive terminal.
-- **GuruRMM PowerShell invocation quirk.** Running `command_type: powershell` fails on PST machines with "-OutputEncoding is not recognized." Use `command_type: cmd` and call `powershell.exe` explicitly within the script body.
+- **GuruRMM command_type — use `powershell` or `shell`, NOT a made-up type (RESOLVED 2026-06-12).** The old advice here ("use `command_type: cmd` and call powershell.exe") was wrong on two counts: (1) the `-OutputEncoding` PowerShell failure it worked around is fixed in the agent (it sets `[Console]::OutputEncoding` inline, so `command_type: "powershell"` works on PST machines); (2) the agent's `CommandType` enum only accepts `shell`, `powershell`, `python`, `script`, `claude_task` (+ alias `cmd` → shell, added 2026-06-12). A command with an **unknown** `command_type` (e.g. the bare `cmd` before the alias) fails the agent's whole-message JSON parse and is **silently dropped** — no ack, no result — which looks exactly like a network black-hole and cost a long mis-diagnosis. Always use `powershell` (runs powershell.exe, UTF-8 fixed) or `shell`/`cmd` (runs cmd.exe). The agent now also NAKs an unparseable command so it fails fast instead of black-holing.
 - **Machine cert template (PEACEFULSPIRIT-PST-SERVER-CA / Machine template).** `msPKI-Certificate-Name-Flag` was changed from `0x18000000` to `0x1` (ENROLLEE_SUPPLIES_SUBJECT) on 2026-05-11. This is a domain-wide template change. New machine certs will use the CSR Subject/SAN rather than the submitting machine's AD DNS identity. RRAS UserAuthProtocolAccepted now includes Certificate (added 2026-05-11).
 - **OneDrive KFM on WSE folder-redirected profiles.** Machines formerly managed by Windows Server Essentials had WSE-specific non-standard GUID variants in User Shell Folders (different from standard Known Folder GUIDs). Direct HKU writes alone do not clear the shell's internal known folder policy state — `SHSetKnownFolderPath` must be called with `flags=0` (not 0x4000) in user session context. If KFM still fails after registry cleanup, wipe the profile and redeploy with per-machine OneDrive (`/allusers`).
 - **pst-admin vs sysadmin distinction.** `pst-admin` is a domain user (in WseRemoteAccessUsers, VPN-eligible). `sysadmin` is domain admin. Many early session failures were caused by using pst-admin credentials for domain admin operations.