claudetools/session-logs/2026-07/2026-07-06-mike-bug024-musl-linux-agent-fix.md

User

• User: Mike Swanson (mike)
• Machine: GURU-5070
• Role: admin

Session Summary

Investigated a report that Howard "broke all the Linux RMM agents" and found it was already a tracked issue — BUG-024 (GuruRMM). Root cause: the build host .30 was upgraded to Ubuntu 26.04 / glibc 2.43, and build-linux.sh compiles the agent against the host toolchain, so Linux agent binaries >=0.6.77 require GLIBC_2.39. Separately, a 2026-07-04 update-scanner fix enabled Linux update offers for the first time, so every reachable older-distro Linux agent pulled the update and crash-looped on GLIBC_2.39 not found (silent — the crash-looping agents still showed "online" until a status-reconciliation fix). Howard had mitigated it by quarantining Linux binaries out of /var/www/gururmm/downloads and reviving the ACG-internal agents (ix/Jupiter); two client agents remained down (Lonestar, Scileppi), and the permanent fix decision (musl static vs old-glibc build container) was parked on Mike.

Mike chose musl static, and I took over the fix. Restructured agent/Cargo.toml so non-Windows targets use rustls (pure-Rust, no OpenSSL) via cfg(not(windows)) — OpenSSL cannot link cleanly/static on musl, rustls can, and macOS already used this config; Windows kept native-tls/schannel for Win7/2008R2. Updated build-linux.sh to build x86_64-unknown-linux-musl static, set CC_x86_64_unknown_linux_musl=musl-gcc (ring needs a musl C compiler), and added an objdump/readelf gate that refuses to publish if the binary has any GLIBC_* refs or a dynamic interpreter. Installed musl-tools on .30 (the one missing dep). Built + verified in an isolated worktree: static-pie, ldd "statically linked", GLIBC refs = 0, runs (gururmm-agent 0.6.79). Committed to branch fix/linux-musl-glibc-floor, pushed to gururmm.git, and handed the fleet deploy (merge -> rebuild image -> un-quarantine) back to Howard (he owns the build host + quarantine + has in-flight branches).

Recovered Lonestar's down agent as a field test. It's a Docker container (bfcfbc739d23, Debian 12 image) on an Unraid box (192.168.120.177, via VPN), crash-looping on the exact BUG-024 glibc error. docker cp was unusable on that btrfs host (it chokes on the /dev/kvm and libvirt-sock special mounts), and a stale libvirt-sock mount (libvirtd was off) blocked container start until a concurrent session fixed libvirtd. I wrote the musl binary directly into the container's RW-layer btrfs subvolume (which preserved the container's device identity — no re-enroll/duplicate) and started it: the agent came up healthy, online in RMM as v0.6.79. Stashed a persistent copy of the binary on the box.

Attempted Scileppi's down agent (SL-SERVER) but hit a credential wall. Established that the box is a Synology RS2212+ at 192.168.242.5 (reached via a Scileppi VPN Mike connected — GURU-5070 has no direct route, and the neighboring Mac relay was a dead end because Mac-mini-2 runs on an iPhone USB tether, not the office LAN). SSH auth failed: the vaulted sysadmin password is stale (rejected), my GURU-5070 key isn't authorized, and admin/root also failed — I stopped to avoid Synology auto-block. Also diagnosed the Mac-mini-2 tether as the root cause of that user's chronic "can't reach the server" problems. Howard is going on-site to handle the Mac hotspot and can grab the current SL-SERVER credential in the same trip.

Key Decisions

• musl static over an old-glibc build container (Mike's call) — removes the glibc floor entirely, so the agent runs on any Linux distro regardless of the build host's glibc.
• Linux -> rustls (ring), not native-tls/OpenSSL — OpenSSL can't link static/clean on musl; rustls is pure-Rust and mirrors the existing macOS config. Windows keeps native-tls unchanged (Win7/2008R2 compat).
• Built in an isolated git worktree on .30 (not the production /home/guru/gururmm checkout) to avoid disturbing the webhook.
• Lonestar fix via direct RW-layer FS write, not docker cp (broken on btrfs) and not a container recreate (which would lose the device identity and re-pull the bad image binary).
• Stopped credential guessing on the Synology to avoid tripping DSM auto-block.
• Handed the fleet deploy to Howard — he owns the build host, the quarantine mechanism, and has in-flight branches; two people shouldn't both touch the build pipeline.

Problems Encountered

• Pushed the musl branch to the wrong repo (guru-rmm.git with a hyphen vs the canonical gururmm.git) — the build host couldn't fetch it. Re-pointed the submodule origin to gururmm.git and re-pushed. (Stray branch on guru-rmm.git still needs cleanup.)
• musl build failed initially: ring's build script needs x86_64-linux-musl-gcc, which was missing. Installed musl-tools via sudo -S with the vaulted guru password, and set CC_x86_64_unknown_linux_musl=musl-gcc.
• docker cp on the Lonestar btrfs box failed on the /dev/kvm device mount (and earlier the libvirt-sock dir mount blocked start). Bypassed by writing the binary directly to the container's RW subvolume once libvirtd was fixed by another session.
• Repeated a past mistake early — treated "can't reach X from here" as "X is down" — but corrected it by verifying via on-segment/RMM paths (same lesson as the UDM earlier this session).
• Scileppi Mac relay was a dead end (Mac on iPhone tether); resolved when Mike connected a Scileppi VPN, giving GURU-5070 direct access.

Configuration Changes

• guru-rmm submodule, branch fix/linux-musl-glibc-floor (commits 2471946, cd81d44, pushed to gururmm.git):
  • agent/Cargo.toml — TLS restructured: cfg(not(windows)) -> rustls (rustls-tls-native-roots); cfg(windows) -> native-tls (tokio-tungstenite/reqwest native-tls + native-tls = "=0.2.11"); macOS keeps blocking + plist via feature union.
  • deploy/build-pipeline/build-linux.sh — TARGET=x86_64-unknown-linux-musl, rustup target add, CC_x86_64_unknown_linux_musl=musl-gcc + AR, build gate (objdump GLIBC + readelf INTERP must be 0), deploy from the musl target dir.
• Build host .30 — apt-get install musl-tools (installed). Left a /tmp/musl-build worktree (cleanup pending).
• Lonestar Unraid (192.168.120.177) — wrote the musl 0.6.79 binary into container bfcfbc739d23's RW-layer subvolume (/usr/local/bin/gururmm-agent) and restarted it; stashed a persistent copy at /mnt/user/appdata/gururmm/gururmm-agent-musl-0.6.79.
• No main-repo file changes beyond this session log.

Credentials & Secrets

• No new credentials created. Used from vault: infrastructure/gururmm-server.sops.yaml (guru SSH key auth to .30 + the guru password for sudo -S); clients/lonestar-electrical/unraid-server.sops.yaml (Unraid root, via plink).
• STALE CRED FLAG: clients/scileppi-law/rs2212.sops.yaml — the sysadmin password is rejected by SSH on SL-SERVER (192.168.242.5). Needs updating once Howard obtains the current login on-site. My GURU-5070 key is NOT authorized there; admin/root also failed.

Infrastructure & Servers

• Build host .30 = gururmm = 172.16.3.30, Ubuntu 26.04 / glibc 2.43, Rust build host, repo /home/guru/gururmm, web root /var/www/gururmm/downloads (Linux binaries quarantined). SSH guru@172.16.3.30 (key). objdump+cc present; musl-tools now installed; cmake still absent (not needed — reqwest uses ring, not aws-lc).
• Down/recovered Linux agents: ix (D2TESTNAS, CloudLinux 9), Jupiter (Debian 12 docker), gururmm, GURU-KALI — fine/revived. Lonestar bfcfbc739d23 (Debian 12 container on Unraid "Tower" 192.168.120.177) — RECOVERED to v0.6.79. Scileppi SL-SERVER — Synology RS2212+ at 192.168.242.5 (stale vault IP 172.16.1.59), DSM on 5000/5001, agent v0.5.1 down ~37h.
• Scileppi Mac-mini-2 (RMM id 1386d9fd-ac16-423c-ada0-5abad5b61838) — running on an iPhone USB tether (192.0.0.2, gw 192.0.0.1) instead of the office LAN; root cause of its chronic server-connection failures. Howard on-site.
• AD2 re-enrolled during the broader session: new RMM id 4b9fe588-2a86-4855-912e-d90cf5f5e3e4.

Commands & Outputs

• Build (on .30, in worktree): CC_x86_64_unknown_linux_musl=musl-gcc AR_x86_64_unknown_linux_musl=ar cargo build --release --target x86_64-unknown-linux-musl.
• Verify (gate): ldd BIN -> "statically linked"; objdump -T BIN | grep -c GLIBC_ -> 0; readelf -l BIN | grep -c INTERP -> 0; file BIN -> "static-pie linked"; BIN --version -> gururmm-agent 0.6.79. musl binary sha256 538d8bc77a596192be89c4481db3cced12aa81c03cfc0dd1fcb1679f0d352d7f.
• Lonestar recovery: derive RW subvol via cat /var/lib/docker/image/btrfs/layerdb/mounts/<CID>/mount-id; cp musl-bin <subvol>/usr/local/bin/gururmm-agent; docker start. Result: running, restarts=0, logs show server-connected/heartbeat/inventory, RMM last_seen current, v0.6.79.
• SL-SERVER auth: sysadmin/vaulted-pw -> "password not accepted"; admin -> "Access denied"; root -> connection closed (root SSH disabled); key auth -> denied.

Pending / Incomplete Tasks

• Howard (deploy): merge fix/linux-musl-glibc-floor -> main -> webhook rebuilds musl (gate-checked) -> un-quarantine Linux downloads in /var/www/gururmm/downloads.
• Scileppi SL-SERVER: blocked on the stale credential — Howard to get the current SSH login on-site (or add my key), then it's the same ~2-min fix (container-vs-native check + musl binary + restart). Update the vault entry.
• Scileppi Mac-mini-2 hotspot: Howard on-site (confirm real Ethernet is connected so it stops falling back to the iPhone tether).
• Cleanup: remove /tmp/musl-build worktree on .30; delete the stray fix/linux-musl-glibc-floor branch on the wrong guru-rmm.git repo.
• Do NOT recreate the Lonestar container until Howard's fixed image ships (fix is in the RW layer; recreate would re-pull the bad image binary).
• Carryover from earlier in the session: Dataforth #32489 John reply (held) + billing (waiting on Howard's line items landing; Dataforth has a 28h prepaid block).

Reference Information

• BUG-024 details: projects/msp-tools/guru-rmm/docs/FEATURE_ROADMAP.md (entry) + Howard's session-logs/2026-07/2026-07-05-howard-* logs.
• Branch fix/linux-musl-glibc-floor HEAD cd81d44 (parent 2471946) on gururmm.git.
• Coord messages to HOWARD-HOME: 20daedf9 (fix ready), cf35bf16 (Lonestar tested), plus the musl-decision relays.
• Lonestar: Unraid 7.1.4, container gururmm-agent/bfcfbc739d23, image git.azcomputerguru.com/azcomputerguru/gururmm-agent:latest, config /mnt/user/appdata/gururmm/config.toml.