claudetools/session-logs/2026-06/2026-06-22-howard-gururmm-dispatch-hang-fix-guruscan-paused.md

GuruRMM fleet dispatch-hang fix + GuruScan Emsisoft test (paused) — 2026-06-22

User

• User: Howard Enos (howard)
• Machine: Howard-Home
• Role: tech

Session Summary

Resumed the GuruScan multi-engine malware-scanner verification on test VM DESKTOP-MS42HNC (the open item from yesterday: run the heavy Emsisoft full /f=C:\ engine to confirm full scan + full removal + automated lifecycle, after HitmanPro's 36-threat lifecycle was already proven). Updated this machine's guru-scan submodule from a 3-week-old 2f8fbcd to the hardened fb09102, confirmed the test VM was back online, and launched the automated scan-one Emsisoft harness — which died instantly at the first step with [ERROR] Dispatch failed.

Tracing that failure uncovered a production incident much larger than the scan: the live GuruRMM server (172.16.3.30:3001) was hanging on every POST /api/agents/:id/command (30s+ timeouts) while GET worked. Reproduced on two independent online agents — command dispatch was down fleet-wide. Root-caused it in the deployed main code: AgentConnections::send_to did a blocking tx.send(msg).await on a bounded (cap 100) per-agent mpsc channel; a black-holed/half-open agent socket stops its WS writer draining the channel, it fills, and the send blocks forever. send_command holds state.agents.read().await across that await, so the next agent (re)connect's .write().await starves tokio's write-preferring RwLock, queuing all subsequent dispatches behind it — one dead socket wedged the whole fleet. The recovery path ("evict non-delivering connections", 7c578fd) had been reverted in 80df458, leaving no escape hatch.

Before touching anything, preserved the "HELD" guru-rmm working tree (detached HEAD at the SPEC-030 uninstall-engine commit 4ca1d06 plus 8 uncommitted files of unrelated DoS-hardening

• dashboard + docs work) onto branch wip/held-2026-06-22 (commit 3a6277a) and pushed it to origin for durable backup — nothing lost. Then, with Howard's clearance to follow the RMM update rules, fixed send_to to use non-blocking try_send (a full/closed channel returns "not delivered"; the command stays persisted and is re-offered by redispatch_pending_commands on reconnect and the reaper requeue_undelivered_commands). Verified with cargo check (via the gururmm-build skill), committed 9dae20c, merged to main, and pushed — which triggers the webhook build+restart on .30. Dispatch recovered ~2.5 min later; verified a command ran end-to-end on a healthy second agent (DESKTOP-QNP3ON5) in ~5s.

The test VM itself then needed recovery: it went silent at the exact moment of the server restart and didn't reconnect (its online was stale grace-period). Howard restarted the GuruRMM agent service on it; the agent reconnected and, after draining the redispatched backlog, executed commands cleanly (~1s round-trips). The ~480 junk commands created by the hung-dispatch attempts self-drained to failed via the reaper (won't re-run). Relaunched the real scan-one Emsisoft run: setup restored 516 malware samples and the detached, no-cap GuruScan-one scheduled task reached state=Running — the full scan was underway. Howard then rebooted DESKTOP-MS42HNC to test other software, interrupting the scan. Per his direction, the Emsisoft verification is marked IN TEST / paused to resume later; the background harness was stopped, scratch files cleaned, and state saved.

Key Decisions

• Fixed + deployed the server bug rather than just reporting it — Howard explicitly cleared "following the rules for updating the RMM and following Mike's requests do what is needed." The merge-to-main deploy is the documented path (gururmm-build skill) and the only way to unblock; auto-rollback protects a bad binary.
• Minimal hotfix = non-blocking try_send, signature unchanged — kept send_to as async fn so all ~24 callers stay untouched; only the body changed. A black-holed agent now degrades to "command queued" (re-offered on reconnect / by the reaper) instead of wedging the fleet. Did NOT re-implement the reverted eviction (larger change, reverted for a reason) — left as a documented follow-up.
• Preserved the HELD guru-rmm work before any git surgery — branched the dangling SPEC-030 commit + uncommitted edits onto wip/held-2026-06-22 and pushed it, so deploying a clean main-based fix never risked the in-progress work. Did not bundle that unrelated WIP into the production deploy.
• Did not manually fight the junk-command backlog — the server's reaper fails undeliverable commands on its own (pending → failed), so manual per-id cancel was redundant and raced it.
• Marked Emsisoft verification IN TEST / paused — VM rebooted mid-scan for other testing; resume cleanly via the self-restoring harness.

Problems Encountered

• Fleet-wide command-dispatch hang — blocking bounded-channel send in send_to + read-lock held across the await → RwLock writer starvation. Fixed with try_send (9dae20c), deployed, verified end-to-end on a second agent.
• Bash cwd parked in a submodule — an earlier cd projects/msp-tools/guru-rmm persisted, so relative paths failed (guru-scan "not found"). Switched to absolute C:/claudetools/... paths.
• Test VM silent after server restart — DESKTOP-MS42HNC didn't reconnect on its own (laptop VM, sleeps). Howard restarted the agent service; it recovered. A first post-reconnect test command timed out transiently while the agent drained the redispatch backlog, then went healthy.
• Scan interrupted by reboot — Howard rebooted the VM to test other software; the in-flight Emsisoft scan was lost. Marked paused; harness self-restores samples on resume.

Configuration Changes

Modified (guru-rmm submodule, committed + deployed):

• server/src/ws/mod.rs — AgentConnections::send_to now uses non-blocking tx.try_send(msg) instead of tx.send(msg).await; added a detailed comment on the fleet-wide-wedge root cause. Commit 9dae20c on main (pushed → webhook build+deploy).

Preserved (guru-rmm submodule, pushed, NOT deployed):

• Branch wip/held-2026-06-22 (3a6277a on top of 4ca1d06) — the held SPEC-030 uninstall prototype + uncommitted DoS-hardening (cap_field/cap_vec in ws/mod.rs, agents.rs), dashboard edits, docs, script-library/time-date/, migration 060_alert_mutes_agent_id_index.sql.

Submodule pointer:

• guru-scan updated to hardened fb09102 (was 2f8fbcd).
• guru-rmm submodule moved from the HELD detached state to main@9dae20c.

Memory (main repo):

• .claude/memory/project_guruscan_in_test_paused.md (new) + index line.
• .claude/memory/project_gururmm_dispatch_hang_fix.md (new) + index line.

Credentials & Secrets

• None created or discovered. GuruRMM API creds read from vault infrastructure/gururmm-server.sops.yaml (credentials.gururmm-api.admin-email / admin-password).
• DESKTOP-MS42HNC still has Windows Defender RTP + Tamper Protection DISABLED (Howard, at the console, for malware testing) — must be re-enabled during final cleanup.

Infrastructure & Servers

• GuruRMM server — http://172.16.3.30:3001; deploy = merge to gururmm main → Gitea webhook on .30 rebuilds (cargo build --release, SQLX_OFFLINE) + systemctl stop/start gururmm-server, auto-rollback if the new binary won't start. Single server binary (/opt/gururmm/gururmm-server), no beta/prod split for the server. Build log /var/log/gururmm-build-server.log.
• DESKTOP-MS42HNC — agent id 0de89b88-b21d-4647-ab64-96157ba87cc5; client AZ Computer Guru, site Howard-VM; flaky laptop test VM (sleeps/reboots). Has malware sample set restored to C:\Users\Owner\Desktop\malware-samples-master (516 files) + zip in Downloads; Defender disabled.
• DESKTOP-QNP3ON5 — agent id ba173f0c-19e8-488d-834c-1b6f6dfd5699; used as the healthy control agent to verify dispatch recovery (command completed in ~5s).
• GuruScan endpoint paths: C:\GuruScan\ (module), C:\GuruScan\downloads\ (scanner EXEs), C:\ScanLogs\<scanid>\ (logs).

Commands & Outputs

• Verify server compiles the deploy way: bash .claude/skills/gururmm-build/scripts/verify.sh server --check → [OK] server check passed.
• Deploy: merge fix/ws-send-to-nonblocking-dispatch-hang → main, git push origin main (b04caf2..9dae20c).
• Recovery verified: dispatch POST HTTP=200 in ~0.1s (was HTTP=000 30s timeout); Write-Output on QNP3ON5 status=completed in ~5s.
• Resume the paused scan (hands-off, self-restores samples): bash .claude/scripts/guruscan-agent-test.sh DESKTOP-MS42HNC scan-one Emsisoft
• Reaper behavior observed: undeliverable backlog pending → failed ("agent unreachable — server-side reaper"), ~480 commands self-cleared.

Pending / Incomplete Tasks

• [IN TEST / PAUSED] Emsisoft full-scan/removal verification on DESKTOP-MS42HNC — VM rebooted mid-scan. Resume with the harness above; expect Emsisoft to clear well beyond HitmanPro's 36 (incl. the .js droppers HitmanPro ignores). Then confirm results.json total_threats / reboot_required + reboot-cleanup task.
• Clean up DESKTOP-MS42HNC — remove samples + zip + EICAR + test tasks, clear scanner quarantine, re-enable Windows Defender RTP + Tamper Protection.
• guru-scan gitlink in main repo — intentionally bumped to fb09102 this session; held no longer (guru-rmm is unblocked).
• Follow-up (server): the proper black-hole connection eviction / keepalive-drop is still absent (was reverted in 80df458). If a half-open agent recurs (heartbeats online but commands dispatch running → reaper-fail on timeout), finish that work rather than relying on try_send alone. Held DoS-hardening on wip/held-2026-06-22 is unreviewed/unmerged.

Reference Information

• guru-rmm fix commit: 9dae20c on main (pushed to https://git.azcomputerguru.com/azcomputerguru/gururmm.git).
• Preserved held work: branch wip/held-2026-06-22 (3a6277a over 4ca1d06), pushed to origin.
• Related prior commits: 7c578fd (evict non-delivering connections — reverted by 80df458), ca1657b (reaper re-delivers black-holed commands), b93f2ef (durable command-ACK foundation).
• guru-scan hardened commit: fb09102.
• Build model + verifier: gururmm-build skill; projects/msp-tools/guru-rmm/docs/BUILD.md.
• Yesterday's GuruScan hardening log: session-logs/2026-06/2026-06-22-howard-guruscan-hardening.md.