From d033dbe8a2cd23af005d889441a1d1c0f1ccf0be Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Thu, 16 Apr 2026 08:34:53 -0700 Subject: [PATCH] Session log: CI signing pipeline + v0.6.1 release + MSI installer MVP End-to-end automated signing via jsign on Linux build server (SP-authenticated to Azure Trusted Signing). First signed release built through the pipeline. First signed MSI installer using WiX 5 on Windows workstation. Co-Authored-By: Claude Opus 4.6 (1M context) --- .../session-logs/2026-04-16-session.md | 307 ++++++++++++++++++ 1 file changed, 307 insertions(+) create mode 100644 projects/msp-tools/guru-rmm/session-logs/2026-04-16-session.md diff --git a/projects/msp-tools/guru-rmm/session-logs/2026-04-16-session.md b/projects/msp-tools/guru-rmm/session-logs/2026-04-16-session.md new file mode 100644 index 0000000..1f2d71b --- /dev/null +++ b/projects/msp-tools/guru-rmm/session-logs/2026-04-16-session.md @@ -0,0 +1,307 @@ +# GuruRMM Session Log — 2026-04-16 + +Full-day session turning yesterday's working signing pipeline into a production release flow and a first proper installer. + +## Session Summary + +Started the morning with a working-but-manual signing setup (Windows-only, from the user's workstation, one binary at a time). Ended the day with an automated Linux-based CI signing pipeline, a signed v0.6.1 agent deployed end-to-end, and a working signed MSI installer. Next step planned: dedicated Windows VM on Jupiter for production MSI builds. + +## Work Completed (chronological) + +### 1. Deployed yesterday's signed v0.6.0 to production + +- Pulled the unsigned v0.6.0 (`gururmm-agent-windows-amd64-0.6.0.exe`, 2,887,680 B, sha256 `79a868...`) from the CDN, signed locally via `sign.ps1`, pushed back to `172.16.3.30:/var/www/gururmm/downloads/` replacing the unsigned copy. New signed binary sha256 `528d87fb07b5e995445336eeb009a8636b9cf7838e4ef6bb4bd39c87d798af4b`, 2,903,424 B. +- `gururmm-agent-windows-amd64-latest.exe` symlink still points at v0.6.0 so it serves the signed version. +- Unsigned original preserved at `/var/www/gururmm/downloads/gururmm-agent-windows-amd64-0.6.0.exe.bak-unsigned-20260416`. +- Cloudflare edge was caching the old unsigned copy. Added `Cache-Control: public, max-age=300, must-revalidate` to nginx `/downloads/` location so future propagation is ≤5 min. Current stale entry will age out within 2h default CF TTL. + +### 2. CI signing: service principal + jsign on Linux build server + +**Service principal created via `az ad sp create-for-rbac`:** +- Name: `gururmm-build-signer` +- AppId / ClientId: `516d0bdc-5416-4d02-8521-b70e2bb26d29` +- Tenant: `ce61461e-81a0-4c84-bb4a-7b354a9a356d` +- Client secret: `S_R8Q~sxkEbkt69hA12U2J13MQcRVVJUEBE62brz` +- Role: `Artifact Signing Certificate Profile Signer` +- Scope: least-privilege — cert-profile-only (`/subscriptions/e507e953-2ce9-4887-ba96-9b654f7d3267/resourceGroups/gururmm-signing-rg/providers/Microsoft.CodeSigning/codeSigningAccounts/gururmm-signing/certificateProfiles/gururmm-public-trust`) + +**Tooling installed on build server (172.16.3.30, Ubuntu 22.04):** +- .NET SDK 8 from MS apt repo (via packages.microsoft.com — Canonical's apt mirrors were unreachable today for some unrelated IPv6 routing issue, but MS repo works) +- Attempted `dotnet sign` global tool — does NOT work on Linux despite being advertised as cross-platform (kernel32.dll dependency, confirmed issue #711 in dotnet/sign). **Uninstalled.** +- Pivoted to **jsign 7.1** (Java, actually cross-platform, explicitly supports Azure Trusted Signing via `TRUSTEDSIGNING` storetype) +- Java: Eclipse Temurin JRE 21.0.10 installed at `/opt/jre/` (tarball install since apt couldn't fetch `default-jre-headless` due to mirror outage). Symlink `/usr/local/bin/java → /opt/jre/bin/java`. +- jsign 7.1 installed via GitHub release deb to `/usr/bin/jsign`. + +**Config files created on 172.16.3.30:** +- `/etc/gururmm-signing.env` — mode 0600, root-owned. Contains SP credentials + Trusted Signing endpoints: + ``` + AZURE_TENANT_ID=ce61461e-81a0-4c84-bb4a-7b354a9a356d + AZURE_CLIENT_ID=516d0bdc-5416-4d02-8521-b70e2bb26d29 + AZURE_CLIENT_SECRET=S_R8Q~sxkEbkt69hA12U2J13MQcRVVJUEBE62brz + TS_ENDPOINT=https://wus2.codesigning.azure.net + TS_ACCOUNT=gururmm-signing + TS_CERT_PROFILE=gururmm-public-trust + TS_TIMESTAMP_URL=http://timestamp.acs.microsoft.com + ``` +- `/opt/gururmm/sign-windows.sh` — wrapper. OAuth client-credentials to fetch access token (scope `https://codesigning.azure.net/.default`), then jsign with `--storetype TRUSTEDSIGNING --storepass $TOKEN --alias gururmm-signing/gururmm-public-trust --tsmode RFC3161 --tsaurl http://timestamp.acs.microsoft.com --alg SHA-256 --replace`. + +**Critical jsign gotcha:** jsign's default timestamp mode is Authenticode, which fails against Microsoft's ACS timestamp URL with `net.jsign.bouncycastle.cms.CMSException: Malformed content`. Forcing `--tsmode RFC3161` fixes it. Documented in the wrapper. + +**build-agents.sh updated on 172.16.3.30 at `/opt/gururmm/build-agents.sh`:** +- Pre-signing backup at `/opt/gururmm/build-agents.sh.bak-pre-signing` +- Post-build step signs the cross-compiled `.exe` in-place via `sign-windows.sh` +- sha256 computed AFTER signing (so consumers get correct hashes) +- `-latest.exe` and `-latest` (Linux) symlinks updated to the new version + +**Bug found during first real build:** VERSION parsing was empty because sed `\1` backreference got eaten during the heredoc write. Fixed by switching to `VERSION=$(awk -F'"' '/^version/{print $2; exit}' agent/Cargo.toml)` — simpler, no escaping issues. + +### 3. v0.6.1 built + signed + deployed end-to-end + +Added agent file logging via `tracing-appender` — the first code change to land through the signing pipeline: + +**Files changed in `azcomputerguru/gururmm` repo (commit b5bc068 + subsequent):** +- `agent/Cargo.toml` — bumped to 0.6.1, added `tracing-appender = "0.2"` +- `agent/src/main.rs` — added `mod logging;`, replaced inline `tracing_subscriber::fmt().init()` with `let _log_guard = logging::init();` +- `agent/src/logging.rs` — new file. Per-platform log path (`C:\ProgramData\GuruRMM\agent.log` on Windows, `/var/log/gururmm/agent.log` on Linux, `/Library/Logs/GuruRMM/agent.log` on macOS), daily rotation via `tracing_appender::rolling::daily`, non-blocking writer with guard. Falls back to stdout-only if the dir isn't writable. + +**Build result:** +- Cargo compile: clean (just pre-existing warnings for unused tunnel/updater methods) +- Linux binary: `/var/www/gururmm/downloads/gururmm-agent-linux-amd64-0.6.1` (3,462,920 B) +- Windows binary (signed): `/var/www/gururmm/downloads/gururmm-agent-windows-amd64-0.6.1.exe` (3,014,440 B, sha256 `0b8acee8cd670f35d70d4b148f3d77c857b4a261ea0f3082712a3551531841cd`) +- Full cert chain verified via signtool from Windows workstation: Microsoft Identity Verification Root → Code Signing PCA 2021 → CS AOC CA 03 → Arizona Computer Guru LLC +- Timestamp: 2026-04-16 07:57:13 UTC +- `-latest.exe` symlink → v0.6.1, serving the signed binary +- `gururmm` self-hosted agent on the build server picked up v0.6.1 immediately (the build script restarts its own service at the end). AD2 and DESKTOP-0O8A1RL still on v0.6.0 as of end-of-session; will auto-update on next poll. + +**Gitea webhook didn't fire on push:** The webhook handler service is running on 172.16.3.30:9000 (`gururmm-webhook.service` listening 127.0.0.1:9000) but nothing called it after `git push`. Likely Gitea doesn't have the webhook configured on the repo, or it's configured but pointing wrong. Manually triggered build via `sudo /opt/gururmm/build-agents.sh`. **Open item — configure Gitea webhook.** + +### 4. MSI installer MVP (WiX 5) + +**Decision:** Skip MSIX (Windows 10+ only, sandboxed, overkill for background agent). Ship MSI first. + +**Discovered:** WiX 7 now requires OSMF (Open Source Maintenance Fee) EULA — may cost money commercially. Stepped back to WiX 5.0.2 which has no such restriction. + +**Discovered:** WiX 5 officially Windows-only despite .NET tooling. Tried on Linux build server anyway — WiX 5 warns "behavior undefined" and actually errors (`WIX0389: The Directory/@Name attribute's value is not a relative path` on a valid Linux path). Pivoted to Windows. + +**Built + signed on Windows workstation:** +- Installed .NET SDK 8.0.420 via winget (`winget install --id Microsoft.DotNet.SDK.8 -e`) +- Installed WiX 5.0.2 (`dotnet tool install --global wix --version 5.0.2`) +- Created wxs at `projects/msp-tools/guru-rmm/installer/gururmm.wxs` +- Built: `wix build gururmm.wxs -arch x64 -o gururmm-agent-0.6.1.msi` → 1.16 MB MSI +- Signed via `C:\tools\trusted-signing\sign.ps1` → full chain verifies + +**Test install on workstation (`msiexec /i ... /qn`):** +- Binary deployed to `C:\Program Files\GuruRMM\gururmm-agent.exe` +- `C:\ProgramData\GuruRMM\` created +- Apps & Features entry: "GuruRMM Agent, 0.6.1, Arizona Computer Guru LLC" +- Installed binary signature preserved (same sha256 as source, signtool verifies full chain) +- Uninstall (`msiexec /x ... /qn`) removes Program Files contents, retains ProgramData (correct — that's user data) + +**Phase 2 (deferred):** Service registration via `ServiceInstall`, MSI properties for `SITE_CODE`/`SERVER_URL`/`API_KEY` injection, custom actions for `gururmm-agent install`, firewall rules. + +### 5. Decision: Windows build VM on Jupiter + +User offered to provision a Windows VM on Jupiter for a proper Windows-native build worker. Recommended spec: +- Windows Server 2022 Standard (supported until Oct 2031) +- 4 vCPU, 8 GB RAM, 80 GB disk +- Gen 2 Hyper-V VM if going that route +- Static IP on 172.16.3.0/24 +- Hostname `gururmm-win-build` +- Role: MSI build + install smoke testing + future Windows-only CI tasks + +**VM not yet provisioned.** Hypervisor decision (Unraid QEMU on Jupiter vs dedicated Hyper-V host) still open. User has licenses for Win 10 / 11 / Server 2019 / 2022. + +## Credentials & Secrets + +### Azure Trusted Signing — service principal for CI + +- **Name:** gururmm-build-signer +- **AppId (client_id):** `516d0bdc-5416-4d02-8521-b70e2bb26d29` +- **Tenant id:** `ce61461e-81a0-4c84-bb4a-7b354a9a356d` +- **Client secret:** `S_R8Q~sxkEbkt69hA12U2J13MQcRVVJUEBE62brz` +- **Role:** Artifact Signing Certificate Profile Signer +- **Scope:** cert-profile `/subscriptions/e507e953-2ce9-4887-ba96-9b654f7d3267/resourceGroups/gururmm-signing-rg/providers/Microsoft.CodeSigning/codeSigningAccounts/gururmm-signing/certificateProfiles/gururmm-public-trust` +- **Rotation:** 12 months. Use `az ad sp credential reset --id 516d0bdc-5416-4d02-8521-b70e2bb26d29` +- **Role assignment id:** `87bbe1e6-b189-445c-b812-2eca5775184a` +- **SP deployment:** `/etc/gururmm-signing.env` on 172.16.3.30 (mode 0600, root) + +### Azure Trusted Signing — account + +- **Subscription:** Basic (`e507e953-2ce9-4887-ba96-9b654f7d3267`) +- **Account:** `gururmm-signing` in rg `gururmm-signing-rg`, westus2 +- **Endpoint:** https://wus2.codesigning.azure.net/ +- **Cert profile:** `gururmm-public-trust`, Public Trust, subject `CN=Arizona Computer Guru LLC, O=Arizona Computer Guru LLC, L=Tucson, S=Arizona, C=US` +- **EKU:** `1.3.6.1.4.1.311.97.842500947.753485959.130432357.333604967` +- **Cert lifetime:** 3 days (by design; per-sign rotation) +- **Timestamp URL:** http://timestamp.acs.microsoft.com (RFC3161 mode when using jsign) + +### GuruRMM admin + +- **Public API:** https://rmm-api.azcomputerguru.com +- **Dashboard:** https://rmm.azcomputerguru.com +- **Admin email / password:** `admin@azcomputerguru.com` / `GuruRMM2025` + +### GuruRMM server (172.16.3.30 / Ubuntu 22.04) + +- **SSH:** `guru` / `Gptf*77ttb123!@#-rmm` (vault: `infrastructure/gururmm-server.sops.yaml`) +- **Sudo password:** same as SSH +- **Postgres:** local, db `gururmm`, user `gururmm`, password `43617ebf7eb242e814ca9988cc4df5ad` +- **JWT secret:** `ZNzGxghru2XUdBVlaf2G2L1YUBVcl5xH0lr/Gpf/QmE=` +- **MariaDB (ClaudeTools):** db `claudetools`, user `claudetools`, password `CT_e8fcd5a3952030a79ed6debae6c954ed` + +### Business identity (for cert recovery / re-verification) + +- **Legal entity:** Arizona Computer Guru LLC +- **D-U-N-S:** 00-566-1506 (005661506) +- **EIN:** 20-5419777 +- **Address:** 7437 E 22nd St, Tucson, AZ 85710 +- **Phone:** +1 520 304 8300 +- **Identity Validation ID:** `03028768-f611-4904-aa58-c755020f436a` + +### Gitea + +- **URL:** https://git.azcomputerguru.com +- **User:** azcomputerguru / `Gptf*77ttb123!@#-git` +- **API token:** `9b1da4b79a38ef782268341d25a4b6880572063f` +- **SSH:** `ssh://git@172.16.3.20:2222` + +## Infrastructure & Files Touched + +### On 172.16.3.30 (GuruRMM / build server) + +| Path | Purpose | +|---|---| +| `/etc/gururmm-signing.env` | SP credentials for jsign (mode 0600, root) | +| `/opt/gururmm/sign-windows.sh` | jsign wrapper — OAuth + TRUSTEDSIGNING storetype | +| `/opt/gururmm/build-agents.sh` | CI build script, now signs Windows binary in-place | +| `/opt/gururmm/build-agents.sh.bak-pre-signing` | Backup of original build script | +| `/opt/gururmm/webhook-handler.py` | Gitea webhook handler (listens 127.0.0.1:9000) — webhook not wired from Gitea side | +| `/opt/jre/` | Eclipse Temurin JRE 21 (tarball install) | +| `/usr/bin/jsign` | jsign 7.1 (via deb) | +| `/home/guru/.dotnet/tools/wix` | WiX — doesn't work on Linux, leaving installed in case useful later | +| `/etc/nginx/sites-available/gururmm` | Added `Cache-Control: public, max-age=300, must-revalidate` on `/downloads/` | +| `/var/www/gururmm/downloads/gururmm-agent-windows-amd64-0.6.0.exe` | Signed v0.6.0 | +| `/var/www/gururmm/downloads/gururmm-agent-windows-amd64-0.6.0.exe.bak-unsigned-20260416` | Unsigned rollback | +| `/var/www/gururmm/downloads/gururmm-agent-windows-amd64-0.6.1.exe` | Signed v0.6.1 (sha256 `0b8acee8...`) | +| `/var/www/gururmm/downloads/gururmm-agent-linux-amd64-0.6.1` | Linux v0.6.1 | +| `/var/www/gururmm/downloads/gururmm-agent-windows-amd64-latest.exe → ...-0.6.1.exe` | Symlink | +| `/var/www/gururmm/downloads/gururmm-agent-linux-amd64-latest → ...-0.6.1` | Symlink | + +### On user's Windows workstation + +| Path | Purpose | +|---|---| +| `C:\tools\trusted-signing\` | sign.ps1, metadata.json, dlib — from yesterday | +| `C:\Users\guru\.dotnet\tools\wix.exe` | WiX 5.0.2 | +| `C:\Program Files\dotnet\` | .NET SDK 8.0.420 (installed today via winget) | +| `D:\work\gururmm\` | Fresh git clone of `azcomputerguru/gururmm` — used to push v0.6.1 changes | + +### In this repo (claudetools) + +| Path | Purpose | +|---|---| +| `projects/msp-tools/guru-rmm/signing-attestation/signing-config/sign-windows.sh` | jsign wrapper (pulled from server) | +| `projects/msp-tools/guru-rmm/signing-attestation/signing-config/build-agents.sh` | Build script (pulled from server) | +| `projects/msp-tools/guru-rmm/signing-attestation/signing-config/binaries/` | Local copies of signed test binaries | +| `projects/msp-tools/guru-rmm/installer/gururmm.wxs` | WiX installer definition | +| `projects/msp-tools/guru-rmm/installer/build-msi.ps1` | Build + sign wrapper for MSI | +| `projects/msp-tools/guru-rmm/installer/README.md` | Installer docs | +| `projects/msp-tools/guru-rmm/installer/.gitignore` | Excludes *.msi, *.wixpdb, src/*.exe | + +### In vault repo + +- `services/azure-trusted-signing.sops.yaml` — updated with SP credentials, IV passed status, cert profile details, test sign results. Fully encrypted with age. + +## Important Commands (reference) + +### Sign a Windows binary on the build server + +```bash +sudo /opt/gururmm/sign-windows.sh /path/to/binary.exe "Optional description" +``` + +### Trigger a build manually (when webhook doesn't fire) + +```bash +ssh guru@172.16.3.30 +sudo /opt/gururmm/build-agents.sh +``` + +### Build + sign an MSI on Windows + +```powershell +cd D:\claudetools\projects\msp-tools\guru-rmm\installer +.\build-msi.ps1 -Version 0.6.1 +``` + +### Verify a signed binary + +```powershell +& 'C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x64\signtool.exe' verify /pa /v +``` + +### Rotate the SP credential + +```bash +az ad sp credential reset --id 516d0bdc-5416-4d02-8521-b70e2bb26d29 +# Update /etc/gururmm-signing.env with the new secret +# Update vault services/azure-trusted-signing.sops.yaml +``` + +## Problems Encountered + Resolutions + +| Problem | Resolution | +|---|---| +| `dotnet sign` tool crashes on Linux with `kernel32.dll.so not found` | Not actually cross-platform. Switched to jsign (Java, verified cross-platform). | +| jsign Authenticode timestamp mode fails against ACS URL | Force `--tsmode RFC3161` | +| Build server can't reach Ubuntu apt mirrors (IPv6 routing problem) | Installed Temurin JRE from tarball, jsign from GitHub deb release directly. Not ideal but works. Real fix: fix IPv6 routing on the network, not our problem today. | +| VERSION came out empty in first build | `sed '\1'` backreference eaten through heredoc. Switched to `awk -F'"' '/^version/{print $2; exit}'` | +| WiX 7 requires OSMF EULA | Stepped back to WiX 5.0.2 | +| WiX officially Windows-only despite .NET tooling | Build MSI on Windows workstation; ship to Linux server for hosting | +| Gitea webhook didn't trigger build on `git push` | Manually ran build-agents.sh; webhook config is an open item | +| Cloudflare edge cached old unsigned v0.6.0 | Added nginx `Cache-Control` header for future builds; waited/manual purge for this one | +| `Get-AuthenticodeSignature` returned empty from bash-spawned PowerShell | PS-from-bash quirk; signtool verify from direct invocation works | + +## Pending / Open Items + +1. **Configure Gitea webhook** on `azcomputerguru/gururmm` repo to POST to `http://172.16.3.30:9000/webhook/build` (or whatever the handler expects). Per the existing `/opt/gururmm/webhook-handler.py`, it listens on 127.0.0.1:9000 with an optional `X-Gitea-Signature` HMAC-SHA256 using `WEBHOOK_SECRET=gururmm-build-secret` (env var default). Need to either expose this externally (nginx proxy) or configure Gitea's outbound webhook to hit an internal path. + +2. **Provision Jupiter Windows VM** per spec above (Server 2022, 4 CPU, 8 GB, 80 GB). Awaiting user decision on hypervisor (Unraid QEMU vs Hyper-V). + +3. **Draft `setup-windows-build-worker.ps1`** once VM is up — installs .NET SDK, WiX 5, Rust + Windows target, VS Build Tools, OpenSSH Server, Gitea runner, Trusted Signing tooling. + +4. **MSI Phase 2** — service registration, site-code injection, agent install custom action. Best done on the Jupiter VM once up. + +5. **Agent fleet update to v0.6.1** — AD2 and DESKTOP-0O8A1RL still on v0.6.0 at end-of-day. Should auto-update on next poll; verify. + +6. **Cloudflare cache for the v0.6.0 swap** — may still be serving stale unsigned binary to some edge regions until ~2h TTL expires. Non-urgent but manual purge via CF dashboard would accelerate. + +7. **Carry-over from yesterday:** + - Tunnel status 403 bug (server/src/db/tunnel.rs) + - Drop `test_records_dedup_bak_20260415` on Dataforth DB after regressions verified + - Cleanup scratch `_*.js` scripts in `C:\Shares\testdatadb\database\` on AD2 + +## Key Reference URLs + +- Public API: https://rmm-api.azcomputerguru.com +- Download listing: https://rmm-api.azcomputerguru.com/downloads/ +- Azure Portal (subscription): https://portal.azure.com/#@/resource/subscriptions/e507e953-2ce9-4887-ba96-9b654f7d3267/resourceGroups/gururmm-signing-rg/providers/Microsoft.CodeSigning/codeSigningAccounts/gururmm-signing/overview +- jsign docs: https://ebourg.github.io/jsign/ +- WiX v5 docs: https://wixtoolset.org/docs/ + +## Commits This Session + +- claudetools `2937c29` — fix VERSION parsing in build-agents.sh (awk, not sed) +- claudetools `148ac75` — installer MVP (WiX wxs, build script, README) +- claudetools `fdd0bb0` — sign-windows.sh + updated build-agents.sh +- gururmm `b5bc068` — agent v0.6.1 with tracing-appender file logging +- vault `8826292` — SP credentials for build-signer +- vault `1df2f83` — cert profile + first test sign details + +## Success Metrics + +- **First successful test sign on Linux build server:** 2026-04-16 14:39 UTC +- **First end-to-end signed release via CI pipeline:** v0.6.1 at 2026-04-16 14:57 UTC +- **First signed MSI:** 2026-04-16 15:15 UTC (gururmm-agent-0.6.1.msi, 1.16 MB) +- **Full Microsoft cert chain validates** through signtool from Windows workstation for both .exe and .msi +- **Billing impact:** Trusted Signing Basic ~$9.99/mo + per-signature fees (fractional cents each). SP creation, cert profile creation, jsign — all free.