claudetools/session-logs/2026-06-02-mike-bsod-detection-and-pipeline.md

User

• User: Mike Swanson (mike)
• Machine: GURU-5070
• Role: admin

Session Summary

GURU-5070 bluescreened, which seeded a two-part session: forensically determine whether GuruConnect caused the crash, and build the long-planned BSOD detection feature for GuruRMM. The crash was identified as VIDEO_TDR_FAILURE (0x116) in nvlddmkm.sys (NVIDIA driver TDR), confirmed by running cdb !analyze -v on the minidump as SYSTEM through the local RMM agent. GuruConnect was cleared on three independent grounds: it was not running at crash time (only agent.toml present, untouched ~7.5h prior), a user-mode app cannot raise a kernel 0x116, and the dump names nvlddmkm.sys on the System process. Root cause: NVIDIA driver 32.0.15.9201 on the new RTX 5070 Ti Laptop GPU; one-off (only event in 21 days).

The BSOD detection feature was taken from spec to production. A shape-spec was written (specs/bsod-detection/), then implemented across agent and server: a Windows-only agent/src/bsod.rs that polls C:\Windows\Minidump, parses the kernel dump header (the minidump crate only reads Breakpad MDMP, so the agent reads DUMP_HEADER64/DUMP_HEADER32 at fixed offsets — bugcheck code @0x38, 4 params @0x40/48/50/58, FILETIME @0xFA8), cross-references the System event log for the Report Id/faulting driver, dedups via a bsod-seen.json watermark, and sends a new AgentMessage::BsodEvent. Server-side: migration 048_bsod_events.sql, db/bsod_events.rs, and a ws/mod.rs handler that inserts the row and raises an alert. Code Review (APPROVE-WITH-NITS) found a real missed-alert-on-boot bug (watermark persisted before send) and a non-atomic watermark write; both were fixed. The offset parser was empirically validated against the real 0x116 dump (every field matched cdb exactly). Severity was set to always-Critical per Mike (overriding the roadmap's graduated rule).

The feature was merged to main (0ec55cf) and the agent versioned to 0.6.51. Two infrastructure gaps surfaced during rollout. First, the build pipeline tagged all new builds stable by default — Mike clarified this was a bug (he wanted agents to default to the stable channel, not builds classified stable). The Windows/Linux build scripts were fixed to default new builds to beta (macOS already did this); stable is now an explicit promote step. Second, the Gitea webhook builds only agents, never the server — build-shared.sh detected the server change and bumped the version, but webhook-handler.py never dispatched build-server.sh. The server (0.3.37, migration 048) had to be built and deployed by hand via build-server.sh through the server's own root RMM agent.

Task 7 verification passed end-to-end on GURU-5070's real 0x116: clearing the watermark seen-list and restarting the agent produced exactly one bsod_events row (code 278, all params matching cdb, correct dump path/sha) and one Critical alert, correctly deduped. The Windows fleet had largely auto-updated to 0.6.51 before the beta re-tag took effect (the soak was effectively lost for already-updated agents); since 0.6.51 was verified and rollback was impossible (the 0.6.50 binary was cleaned up), Mike chose to promote 0.6.51 to stable and converge the fleet.

Finally, the webhook server-build gap was closed: build-server.sh got a server/ change-gate (with a last-built-commit-server marker) plus binary backup + auto-rollback, and webhook-handler.py now dispatches it alongside the agent builds. Deployed live and validated — the gate correctly skips on no-change.

Key Decisions

• Minidump parsing by fixed DUMP_HEADER64 offsets, not the minidump crate — the crate only parses Breakpad MDMP, not Windows kernel PAGEDU64 dumps. Offsets validated against the real dump (sig PAGEDU64, code@0x38=0x116, params matching cdb).
• Always-Critical severity (Mike) instead of the roadmap's 1/24h=warning, >=2=critical. The 24h count is retained as informational text in the alert ("N BSOD in 24h").
• Dedup by dump sha256 (survives re-reads/re-enroll) rather than Report Id; server has a unique index (agent_id, dump_sha256) + alert dedup_key as backstop.
• First-run watermark baselines existing dumps as seen and alerts on none — suppresses historical crashes on a fresh install.
• Build channel: new builds default to beta — the stable-by-default tagging was a bug; promotion to stable is now explicit. Distinct from agents defaulting to the stable channel (correct, unchanged).
• Promote 0.6.51 to stable / converge the fleet — beta-first soak was lost for already-updated agents; 0.6.51 was verified and rollback impossible, so converging was the right call.
• Wire server builds into the webhook with a change-gate + backup/rollback — auto-deploying the server without those would rebuild on every push and risk the BUG-003 no-rollback outage.
• GURU-5070 left on the beta channel as the permanent canary (now meaningful, since builds default beta going forward).

Problems Encountered

• Server's RMM agent appeared to "not see" /var/www/downloads — actually the wrong path; the real dir is /var/www/gururmm/downloads (DOWNLOADS_DIR env override). The agent runs as root and can read/write it fine; the systemd sandbox bites only on mount observations / non-ReadWritePaths writes.
• No sshpass on Windows GURU-5070 — the ix-server memory's sshpass note doesn't apply here. plink/pscp are available (C:\Program Files\PuTTY\). Ultimately used the server's own root RMM agent instead of SSH.
• pgrep -f 'build-server.sh' self-matched the RMM command's own shell text, causing false "already running" aborts. Resolved by launching unconditionally and relying on internal locking.
• Beta re-tag raced the auto-update — the Windows fleet auto-updated to 0.6.51 during the ~20 min between stable publish (02:30 UTC) and the re-tag. Mitigated by promoting 0.6.51 to stable and converging.
• Server never auto-deployed — webhook builds agents only; build-server.sh is manual. Deployed by hand, then wired into the webhook to prevent recurrence.
• Code Review caught SF-1 (watermark persisted before WS send → alert dropped if disconnected at boot, the feature's primary scenario) and SF-2 (non-atomic watermark write). Both fixed before merge.

Configuration Changes

Created (GuruRMM submodule):

• specs/bsod-detection/{plan,shape,references,standards}.md
• agent/src/bsod.rs
• server/migrations/048_bsod_events.sql
• server/src/db/bsod_events.rs

Modified (GuruRMM submodule):

• agent/src/transport/mod.rs, agent/src/transport/websocket.rs — BsodEvent message
• agent/src/main.rs, agent/src/service.rs — spawn detector (both entry points)
• agent/Cargo.toml — version 0.6.50 -> 0.6.51 (also CI auto-bump)
• server/src/db/mod.rs, server/src/ws/mod.rs — register module + handler (always-Critical)
• deploy/build-pipeline/build-windows.sh, build-linux.sh — default new builds to beta
• deploy/build-pipeline/build-server.sh — change-gate + marker + backup/rollback; log to /var/log/gururmm-build-server.log
• deploy/build-pipeline/webhook-handler.py — dispatch build-server.sh (SERVER entry; PLATFORMS + [SERVER])
• docs/FEATURE_ROADMAP.md — issue #10 Agent/Server bullets checked, notes added

Created (ClaudeTools repo):

• .claude/memory/feedback_gururmm_build_channel_default.md (+ MEMORY.md index)
• Updated .claude/memory/reference_gururmm.md (downloads dir, channel control, root-agent access, plink) + MEMORY.md index

Live server changes (172.16.3.30):

• /opt/gururmm/gururmm-server -> v0.3.37 (migration 048 applied)
• /opt/gururmm/webhook-handler.py -> server-build dispatch version (+ systemctl restart gururmm-webhook)
• /opt/gururmm/build-server.sh -> gated/rollback version
• /opt/gururmm/last-built-commit-server -> seeded to faf6b27
• /var/www/gururmm/downloads/gururmm-agent-windows-*-0.6.51.exe.channel -> beta then stable (promote)

Credentials & Secrets

• No new secrets created. Used existing vault entries:
  • infrastructure/gururmm-server.sops.yaml — credentials.gururmm-api.admin-email/admin-password (RMM API JWT); credentials.username=guru / credentials.password (SSH, sudo = SSH pw); host=172.16.3.30, port=22.

Infrastructure & Servers

• GuruRMM server (Saturn): 172.16.3.30 — API :3001, coord API :8001, webhook handler 127.0.0.1:9000, Postgres (DATABASE_URL via process environ), downloads dir /var/www/gururmm/downloads, repo /home/guru/gururmm, pipeline /opt/gururmm/. OS Ubuntu. Runs its own GuruRMM Linux agent AS ROOT (hostname gururmm, agent id 5e5a7ebc-95ea-40c8-b965-6ec15d63e157 on this date — resolve live).
• GURU-5070: Mike's box; the crashed machine; agent id c043d9ac-4020-4cab-a5f4-b90213d11e73; on beta channel; NVIDIA RTX 5070 Ti Laptop GPU + Intel iGPU; driver 32.0.15.9201.
• Internal Gitea: http://172.16.3.20:3000 (git/API).

Commands & Outputs

• BSOD analysis (as SYSTEM via RMM): cdb -z C:\WINDOWS\Minidump\060126-16718-01.dmp -y srv*C:\symbols*https://msdl.microsoft.com/download/symbols -c "!analyze -v;q" -> FAILURE_BUCKET_ID: 0x116_IMAGE_nvlddmkm.sys, PROCESS_NAME: System.
• Offset validation (read dump header bytes): sig=PAGEDU64, code@0x38=0x00000116, p1=FFFFE28CA1998050, p2=FFFFF8016FD61050, p3=FFFFFFFFC000009A, p4=4, FILETIME@0xFA8 -> 2026-06-01T23:57:03Z.
• Task 7 result: bsod_events 1 row code=278 text=VIDEO_TDR_FAILURE params matching; alert critical | active | BSOD: VIDEO_TDR_FAILURE (0x116) on GURU-5070; dedup held (1 row / 1 alert after restart).
• Server deploy: build-server.sh -> Server build complete: v0.3.37 (4m55s); to_regclass('public.bsod_events')=bsod_events; _sqlx_migrations version 48 success=t.
• Webhook gate test: build-server.sh -> No server/ changes since faf6b27 -- skipping server build (server untouched).
• Channel rollout control: echo beta|stable > /var/www/gururmm/downloads/<binary>.channel.

Pending / Incomplete Tasks

• BSOD Phase 2/3 (deferred, in roadmap issue #10): faulting_module is frequently null (event-log xref doesn't carry the driver) — optional cdb !analyze enrichment; dashboard "Crashes" tab + BSOD in Alerts stream; fetch_bsod_dump on-demand upload; full ~350-entry bugcheck name table (Phase 1 ships a 10-code map).
• Fleet convergence to 0.6.51 in progress (stragglers update as they reconnect).
• Optional future hardening: pre-stop migration validation in build-server.sh (backup/rollback covers the binary, not partial migrations).
• webhook-handler.py.pre-serverbuild.bak left on server as a rollback copy.

Reference Information

• Spec: projects/msp-tools/guru-rmm/specs/bsod-detection/ (plan.md is the source of truth; Tasks 1-6 marked DONE).
• GuruRMM commits: 0ec55cf (feature), a8d336a (CI bump 0.6.51/0.3.37), c1bdc1e/a494f22 (build default-beta), faf6b27 (webhook server-build wiring).
• Parent ClaudeTools bumps: 1f817f5, 13c7ad3 (submodule pointer).
• Coord todo: b2e6b994-bb1e-48ef-b42b-a70d1626a535 (webhook server-build gap) — CLOSED.
• Roadmap: docs/FEATURE_ROADMAP.md "Crash Dump Detection & Analysis (BSOD)" / Gitea issue #10.
• Crash fixture: C:\Windows\Minidump\060126-16718-01.dmp, sha256 0b490e2755b40e5726bac69ed99ec95bac8c52e6bf309b3233fc286f01cc2534, WER Report Id 1c76ccf7-ce26-4eb1-ae1c-ab6ef0adbc17.