claudetools/session-logs/2026-07/2026-07-04-mike-gururmm-tray-policy-deploy.md

Session: GuruRMM policy-controlled tray deployment — diagnose, build, ship, promote

User

• User: Mike Swanson (mike)
• Machine: GURU-5070
• Role: admin

Session Summary

Started from a Windows Security screenshot on GURU-5070: "Part of this app has been blocked … can't confirm who published gururmm-tray.exe." Diagnosed it as Smart App Control (SAC) in enforcement mode (VerifiedAndReputablePolicyState=1), which blocks a trusted parent (the signed agent) from loading an unsigned child. The published tray IS signed (Azure Trusted Signing, CN=Arizona Computer Guru LLC); the blocked file was a stale pre-signing tray dated 2026-05-13. Mike hand-replaced it (elevated). SAC was then found to also be blocking grok.exe, rustc/cargo, sops, openssl — Mike turned SAC off (one-way; needs a Windows reset to re-enable).

A fleet census (113-host sample via /rmm) showed the blast radius was ~0: the tray was never deployed fleet-wide — ~98% of hosts had no tray binary at all; only GURU-BEAST-ROG had a stale unsigned one. That reframed the work: the real requirement (per Mike) is that the tray be pushed to every machine but only shown when a client policy enables it, versioned in lockstep with the agent.

A first inline attempt (fold tray refresh into do_update) was rejected by a 17-agent code review (9 defects). The design was then re-shaped with multi-AI input: Claude's review + Gemini 3.1 Pro (caught "no network in the SYSTEM watchdog") + Grok/xAI (caught "materialize ≠ enable"; they disagreed on ownership, resolved toward agent-stages/watchdog-installs). Wrote a 4-file spec (specs/tray-policy-deploy/) and built the feature across agent + watchdog + tray + server: Tasks 1 (policy gate + default-OFF), 2 (materialize FSM + staging + versioned mutex), 3 (server capability-gate + default-OFF), 5 (observability) + Authenticode verify. Every step compile-verified on Beast (amd64 + i686 + server + tray). Security review clean; a second 17-agent code review found 10 findings — the 5 correctness ones fixed (notably the cross-restart cycle and the SCM-poll lock stall), 3 cleanups fixed, 3 documented follow-ups.

Merged to main and deployed v0.6.77. The build initially failed — but only on the legacy Rust 1.77 variant, which re-resolves deps unpinned and now pulls edition2024 crates (chacha20/rand_core 0.10.1) that 1.77 can't parse (BUG-021 recurrence). Proved it reproduces at the pre-feature base, so NOT caused by the change. Per Mike, disabled the legacy variant (LEGACY_ENABLED=false), preserving existing legacy artifacts at 0.6.76 so 2008R2/Win7 endpoints stay supported. Rebuild succeeded; 0.6.77 shipped + signed. Finally promoted the agent 0.6.77 to stable (flipped .channel files) and promoted the dashboard beta → prod (live at rmm.azcomputerguru.com), both done as guru without sudo.

Key Decisions

• Architecture: agent-staged, watchdog-enforced — the agent (already SYSTEM + network for self-update) downloads/verifies/stages the tray; the SYSTEM watchdog does only the local install + policy-gated launch. Resolves the Gemini-vs-Grok disagreement (keeps network out of the supervisor without a second trust story).
• Materialize is policy-agnostic; only launch is policy-gated (Grok's catch) — the binary lands on every machine regardless of enabled; the icon shows only when a client policy sets it.
• Default OFF at the leaf — flipped system_defaults tray.enabled false; the agent now ENFORCES the gate, so a true default would light up the whole fleet on first update.
• Separate tray_download_url + server-authenticated checksum (not a sidecar, not bundled) — reuses the existing publish pipeline + scanner pattern; fixes the CDN-sidecar supply-chain gap.
• Dropped the revision field — review + security review both flagged it as dead state; single-writer + atomic rename makes it unnecessary.
• Deferred Authenticode publisher-pinning + updater/win_swap dedup + tray_stage URL-allowlist — documented follow-ups; the server-authenticated SHA256 is the implemented primary control.
• Kill the legacy Rust 1.77 variant for now (Mike) — 2008R2 support returns via a separate legacy agent (C++/.NET or clean rewrite), NOT by fighting the 1.77 + edition2024 treadmill. Existing legacy artifacts preserved (frozen at 0.6.76).

Problems Encountered

• SAC blocking everything — Smart App Control enforcing on GURU-5070 blocked tray/grok/rustc/sops. Diagnosed via HKLM:\SYSTEM\CurrentControlSet\Control\CI\Policy\VerifiedAndReputablePolicyState. Mike turned SAC off (Windows Security → App & browser control).
• Local MSVC build broken — post-SAC rustc runs but link.exe env isn't loaded in Git-bash; used Beast (guru@100.101.122.4) via throwaway git worktree + patch + cargo check for all verify.
• git diff excluded new files — patches to Beast missed new .rs files until switching to git add -A && git diff --cached <base>.
• windows 0.58 feature-gates WINTRUST_DATA — hand-declared the C-ABI structs (#[repr(C)]) and passed a raw pointer to the windows-rs WinVerifyTrust binding.
• Legacy 1.77 build fail-closed the whole Windows build — edition2024 deps (chacha20/rand_core 0.10.1). Proved pre-existing (reproduces at base 9de44d3). Disabled via LEGACY_ENABLED=false.
• Git-bash path mangling — $SSH "/opt/gururmm/..." mangled the leading /opt to C:/Program; fixed with MSYS_NO_PATHCONV=1 MSYS2_ARG_CONV_EXCL='*'.
• ClaudeTools push to 172.16.3.20:3000 unreachable — the memory+gitlink commit is local (later reconciled via fleet sync; /sync shows clean).
• sudo needs a password on .30 — did the agent-channel flip + dashboard promote as guru (dirs are guru-owned); dashboard promote couldn't append to the root-owned build log (cosmetic).

Configuration Changes

Agent (guru-rmm, all on branch merged to main):

• NEW agent/src/tray_state.rs, agent/src/tray_stage.rs, agent/src/win_swap.rs, agent/src/authenticode.rs, agent/src/watchdog/tray_install.rs
• MOD agent/src/main.rs (module decls), agent/src/transport/mod.rs + websocket.rs (TrayConfigUpdate fields + desired-state writer), agent/src/watchdog/monitor.rs (gate + FSM + telemetry), agent/src/watchdog/wts.rs (session-based terminate_all + tracked_count), agent/Cargo.toml (Win32_Security_WinTrust)
• MOD tray/Cargo.toml (pinned 0.6.76), tray/src/winsingleton.rs (versioned mutex) Server:
• MOD server/src/policy/merge.rs (default-OFF), config_update.rs (materialize fields + gate), server/src/updates/scanner.rs (tray index), server/src/ws/mod.rs (capability-gated send) Pipeline:
• MOD deploy/build-pipeline/build-shared.sh (tray version lockstep bump), deploy/build-pipeline/build-windows.sh (LEGACY_ENABLED=false + preserve legacy artifacts) Spec/docs:
• NEW specs/tray-policy-deploy/{shape,plan,references,standards}.md
• MOD docs/RMM_THOUGHTS.md (Feature 12 — legacy rewrite) Memory (ClaudeTools):
• NEW .claude/memory/project_gururmm_legacy_disabled.md + MEMORY.md index line

Credentials & Secrets

None newly created or discovered. Tray signing uses the existing Azure Trusted Signing SP (/etc/gururmm-signing.env on the build server, root-readable only) via jsign — unchanged.

Infrastructure & Servers

• Build server: 172.16.3.30 (guru@) — webhook-handler.py orchestrates; build scripts in /opt/gururmm/; logs /var/log/gururmm-build-{windows,server,dashboard,linux}.log; downloads /var/www/gururmm/downloads/; dashboard /var/www/gururmm/dashboard{,-beta}.
• Beast (primary Windows build host): guru@100.101.122.4 (Tailscale); sccache C:\sccache; checkout C:\gururmm. Pluto (fallback): Administrator@172.16.3.36.
• Gitea: guru-rmm pushes to git.azcomputerguru.com (HTTPS, worked); ClaudeTools origin 172.16.3.20:3000 (was unreachable this session).
• Live URLs: dashboard prod https://rmm.azcomputerguru.com, beta https://rmm-beta.azcomputerguru.com; agent API wss://rmm-api.azcomputerguru.com/ws.

Commands & Outputs

• SAC state: Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Control\CI\Policy" | VerifiedAndReputablePolicyState → 1 (enforce); after off → 0.
• Beast verify loop: git add -A && git diff --cached <base> > patch → scp → git worktree add --detach C:\gururmm-verify <base> → git apply → cargo +stable check (amd64 + --target i686-pc-windows-msvc).
• Legacy repro at base: move Cargo.lock aside && cargo +1.77 build --features legacy → feature edition2024 is required.
• Agent promote: for f in /var/www/gururmm/downloads/*0.6.77*.channel; do rm -f "$f"; echo stable > "$f"; done.
• Dashboard promote: MSYS_NO_PATHCONV=1 ssh guru@172.16.3.30 /opt/gururmm/promote-dashboard.sh --confirm → backup dashboard-20260704-204039, prod==beta verified.

Pending / Incomplete Tasks

• Enable the tray for a client: set tray.enabled=true on a client policy — until then it deploys dark (default-OFF, by design). Nothing shows an icon yet.
• 2008R2/Win7 legacy agent rewrite (RMM_THOUGHTS Feature 12) — separate C++/.NET or clean rewrite; do NOT re-enable the Rust 1.77 variant as the fix.
• Documented follow-ups (accepted): Authenticode publisher-pinning; migrate updater::replace_binary onto win_swap (+ MOVEFILE_DELAY_UNTIL_REBOOT); restore updater's HTTPS/domain-allowlist in tray_stage; best-effort desired-state write (C3).
• build-shared.sh tray-lockstep bump won't take effect until /opt/gururmm/build-shared.sh is manually synced (build-shared self-excludes from the auto-sync); cosmetic (tray file named by agent version regardless).
• Harness (GURU-5070): CLAUDETOOLS_ROOT=D:/claudetools (wrong case) breaks log-skill-error.sh.

Reference Information

• Version shipped: agent/server/tray 0.6.77 (server 0.3.93). Legacy frozen at 0.6.76.
• Key commits (guru-rmm main): spec 29b2696; feature 20cb05c/42d4651/b7599fd/9294591; fixes 21097bc; merge f6a4251; version-bump 6b8febc; legacy-disable 84a82a3; RMM_THOUGHTS 23a7bcc. Base 9de44d3.
• Spec: projects/msp-tools/guru-rmm/specs/tray-policy-deploy/.
• Downloads: gururmm-{agent-windows-amd64,agent-windows-x86,tray-windows-amd64,agent-base}-0.6.77.* (stable channel); legacy *-0.6.76.exe preserved.