Session log: CI signing pipeline + v0.6.1 release + MSI installer MVP

End-to-end automated signing via jsign on Linux build server (SP-authenticated to Azure Trusted Signing). First signed release built through the pipeline. First signed MSI installer using WiX 5 on Windows workstation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 08:34:53 -07:00
parent 148ac75a25
commit d033dbe8a2
1 changed files with 307 additions and 0 deletions
--- a/projects/msp-tools/guru-rmm/session-logs/2026-04-16-session.md
+++ b/projects/msp-tools/guru-rmm/session-logs/2026-04-16-session.md
@@ -0,0 +1,307 @@
+# GuruRMM Session Log — 2026-04-16
+
+Full-day session turning yesterday's working signing pipeline into a production release flow and a first proper installer.
+
+## Session Summary
+
+Started the morning with a working-but-manual signing setup (Windows-only, from the user's workstation, one binary at a time). Ended the day with an automated Linux-based CI signing pipeline, a signed v0.6.1 agent deployed end-to-end, and a working signed MSI installer. Next step planned: dedicated Windows VM on Jupiter for production MSI builds.
+
+## Work Completed (chronological)
+
+### 1. Deployed yesterday's signed v0.6.0 to production
+
+- Pulled the unsigned v0.6.0 (`gururmm-agent-windows-amd64-0.6.0.exe`, 2,887,680 B, sha256 `79a868...`) from the CDN, signed locally via `sign.ps1`, pushed back to `172.16.3.30:/var/www/gururmm/downloads/` replacing the unsigned copy. New signed binary sha256 `528d87fb07b5e995445336eeb009a8636b9cf7838e4ef6bb4bd39c87d798af4b`, 2,903,424 B.
+- `gururmm-agent-windows-amd64-latest.exe` symlink still points at v0.6.0 so it serves the signed version.
+- Unsigned original preserved at `/var/www/gururmm/downloads/gururmm-agent-windows-amd64-0.6.0.exe.bak-unsigned-20260416`.
+- Cloudflare edge was caching the old unsigned copy. Added `Cache-Control: public, max-age=300, must-revalidate` to nginx `/downloads/` location so future propagation is ≤5 min. Current stale entry will age out within 2h default CF TTL.
+
+### 2. CI signing: service principal + jsign on Linux build server
+
+**Service principal created via `az ad sp create-for-rbac`:**
+- Name: `gururmm-build-signer`
+- AppId / ClientId: `516d0bdc-5416-4d02-8521-b70e2bb26d29`
+- Tenant: `ce61461e-81a0-4c84-bb4a-7b354a9a356d`
+- Client secret: `S_R8Q~sxkEbkt69hA12U2J13MQcRVVJUEBE62brz`
+- Role: `Artifact Signing Certificate Profile Signer`
+- Scope: least-privilege — cert-profile-only (`/subscriptions/e507e953-2ce9-4887-ba96-9b654f7d3267/resourceGroups/gururmm-signing-rg/providers/Microsoft.CodeSigning/codeSigningAccounts/gururmm-signing/certificateProfiles/gururmm-public-trust`)
+
+**Tooling installed on build server (172.16.3.30, Ubuntu 22.04):**
+- .NET SDK 8 from MS apt repo (via packages.microsoft.com — Canonical's apt mirrors were unreachable today for some unrelated IPv6 routing issue, but MS repo works)
+- Attempted `dotnet sign` global tool — does NOT work on Linux despite being advertised as cross-platform (kernel32.dll dependency, confirmed issue #711 in dotnet/sign). **Uninstalled.**
+- Pivoted to **jsign 7.1** (Java, actually cross-platform, explicitly supports Azure Trusted Signing via `TRUSTEDSIGNING` storetype)
+- Java: Eclipse Temurin JRE 21.0.10 installed at `/opt/jre/` (tarball install since apt couldn't fetch `default-jre-headless` due to mirror outage). Symlink `/usr/local/bin/java → /opt/jre/bin/java`.
+- jsign 7.1 installed via GitHub release deb to `/usr/bin/jsign`.
+
+**Config files created on 172.16.3.30:**
+- `/etc/gururmm-signing.env` — mode 0600, root-owned. Contains SP credentials + Trusted Signing endpoints:
+  ```
+  AZURE_TENANT_ID=ce61461e-81a0-4c84-bb4a-7b354a9a356d
+  AZURE_CLIENT_ID=516d0bdc-5416-4d02-8521-b70e2bb26d29
+  AZURE_CLIENT_SECRET=S_R8Q~sxkEbkt69hA12U2J13MQcRVVJUEBE62brz
+  TS_ENDPOINT=https://wus2.codesigning.azure.net
+  TS_ACCOUNT=gururmm-signing
+  TS_CERT_PROFILE=gururmm-public-trust
+  TS_TIMESTAMP_URL=http://timestamp.acs.microsoft.com
+  ```
+- `/opt/gururmm/sign-windows.sh` — wrapper. OAuth client-credentials to fetch access token (scope `https://codesigning.azure.net/.default`), then jsign with `--storetype TRUSTEDSIGNING --storepass $TOKEN --alias gururmm-signing/gururmm-public-trust --tsmode RFC3161 --tsaurl http://timestamp.acs.microsoft.com --alg SHA-256 --replace`.
+
+**Critical jsign gotcha:** jsign's default timestamp mode is Authenticode, which fails against Microsoft's ACS timestamp URL with `net.jsign.bouncycastle.cms.CMSException: Malformed content`. Forcing `--tsmode RFC3161` fixes it. Documented in the wrapper.
+
+**build-agents.sh updated on 172.16.3.30 at `/opt/gururmm/build-agents.sh`:**
+- Pre-signing backup at `/opt/gururmm/build-agents.sh.bak-pre-signing`
+- Post-build step signs the cross-compiled `.exe` in-place via `sign-windows.sh`
+- sha256 computed AFTER signing (so consumers get correct hashes)
+- `-latest.exe` and `-latest` (Linux) symlinks updated to the new version
+
+**Bug found during first real build:** VERSION parsing was empty because sed `\1` backreference got eaten during the heredoc write. Fixed by switching to `VERSION=$(awk -F'"' '/^version/{print $2; exit}' agent/Cargo.toml)` — simpler, no escaping issues.
+
+### 3. v0.6.1 built + signed + deployed end-to-end
+
+Added agent file logging via `tracing-appender` — the first code change to land through the signing pipeline:
+
+**Files changed in `azcomputerguru/gururmm` repo (commit b5bc068 + subsequent):**
+- `agent/Cargo.toml` — bumped to 0.6.1, added `tracing-appender = "0.2"`
+- `agent/src/main.rs` — added `mod logging;`, replaced inline `tracing_subscriber::fmt().init()` with `let _log_guard = logging::init();`
+- `agent/src/logging.rs` — new file. Per-platform log path (`C:\ProgramData\GuruRMM\agent.log` on Windows, `/var/log/gururmm/agent.log` on Linux, `/Library/Logs/GuruRMM/agent.log` on macOS), daily rotation via `tracing_appender::rolling::daily`, non-blocking writer with guard. Falls back to stdout-only if the dir isn't writable.
+
+**Build result:**
+- Cargo compile: clean (just pre-existing warnings for unused tunnel/updater methods)
+- Linux binary: `/var/www/gururmm/downloads/gururmm-agent-linux-amd64-0.6.1` (3,462,920 B)
+- Windows binary (signed): `/var/www/gururmm/downloads/gururmm-agent-windows-amd64-0.6.1.exe` (3,014,440 B, sha256 `0b8acee8cd670f35d70d4b148f3d77c857b4a261ea0f3082712a3551531841cd`)
+- Full cert chain verified via signtool from Windows workstation: Microsoft Identity Verification Root → Code Signing PCA 2021 → CS AOC CA 03 → Arizona Computer Guru LLC
+- Timestamp: 2026-04-16 07:57:13 UTC
+- `-latest.exe` symlink → v0.6.1, serving the signed binary
+- `gururmm` self-hosted agent on the build server picked up v0.6.1 immediately (the build script restarts its own service at the end). AD2 and DESKTOP-0O8A1RL still on v0.6.0 as of end-of-session; will auto-update on next poll.
+
+**Gitea webhook didn't fire on push:** The webhook handler service is running on 172.16.3.30:9000 (`gururmm-webhook.service` listening 127.0.0.1:9000) but nothing called it after `git push`. Likely Gitea doesn't have the webhook configured on the repo, or it's configured but pointing wrong. Manually triggered build via `sudo /opt/gururmm/build-agents.sh`. **Open item — configure Gitea webhook.**
+
+### 4. MSI installer MVP (WiX 5)
+
+**Decision:** Skip MSIX (Windows 10+ only, sandboxed, overkill for background agent). Ship MSI first.
+
+**Discovered:** WiX 7 now requires OSMF (Open Source Maintenance Fee) EULA — may cost money commercially. Stepped back to WiX 5.0.2 which has no such restriction.
+
+**Discovered:** WiX 5 officially Windows-only despite .NET tooling. Tried on Linux build server anyway — WiX 5 warns "behavior undefined" and actually errors (`WIX0389: The Directory/@Name attribute's value is not a relative path` on a valid Linux path). Pivoted to Windows.
+
+**Built + signed on Windows workstation:**
+- Installed .NET SDK 8.0.420 via winget (`winget install --id Microsoft.DotNet.SDK.8 -e`)
+- Installed WiX 5.0.2 (`dotnet tool install --global wix --version 5.0.2`)
+- Created wxs at `projects/msp-tools/guru-rmm/installer/gururmm.wxs`
+- Built: `wix build gururmm.wxs -arch x64 -o gururmm-agent-0.6.1.msi` → 1.16 MB MSI
+- Signed via `C:\tools\trusted-signing\sign.ps1` → full chain verifies
+
+**Test install on workstation (`msiexec /i ... /qn`):**
+- Binary deployed to `C:\Program Files\GuruRMM\gururmm-agent.exe`
+- `C:\ProgramData\GuruRMM\` created
+- Apps & Features entry: "GuruRMM Agent, 0.6.1, Arizona Computer Guru LLC"
+- Installed binary signature preserved (same sha256 as source, signtool verifies full chain)
+- Uninstall (`msiexec /x ... /qn`) removes Program Files contents, retains ProgramData (correct — that's user data)
+
+**Phase 2 (deferred):** Service registration via `ServiceInstall`, MSI properties for `SITE_CODE`/`SERVER_URL`/`API_KEY` injection, custom actions for `gururmm-agent install`, firewall rules.
+
+### 5. Decision: Windows build VM on Jupiter
+
+User offered to provision a Windows VM on Jupiter for a proper Windows-native build worker. Recommended spec:
+- Windows Server 2022 Standard (supported until Oct 2031)
+- 4 vCPU, 8 GB RAM, 80 GB disk
+- Gen 2 Hyper-V VM if going that route
+- Static IP on 172.16.3.0/24
+- Hostname `gururmm-win-build`
+- Role: MSI build + install smoke testing + future Windows-only CI tasks
+
+**VM not yet provisioned.** Hypervisor decision (Unraid QEMU on Jupiter vs dedicated Hyper-V host) still open. User has licenses for Win 10 / 11 / Server 2019 / 2022.
+
+## Credentials & Secrets
+
+### Azure Trusted Signing — service principal for CI
+
+- **Name:** gururmm-build-signer
+- **AppId (client_id):** `516d0bdc-5416-4d02-8521-b70e2bb26d29`
+- **Tenant id:** `ce61461e-81a0-4c84-bb4a-7b354a9a356d`
+- **Client secret:** `S_R8Q~sxkEbkt69hA12U2J13MQcRVVJUEBE62brz`
+- **Role:** Artifact Signing Certificate Profile Signer
+- **Scope:** cert-profile `/subscriptions/e507e953-2ce9-4887-ba96-9b654f7d3267/resourceGroups/gururmm-signing-rg/providers/Microsoft.CodeSigning/codeSigningAccounts/gururmm-signing/certificateProfiles/gururmm-public-trust`
+- **Rotation:** 12 months. Use `az ad sp credential reset --id 516d0bdc-5416-4d02-8521-b70e2bb26d29`
+- **Role assignment id:** `87bbe1e6-b189-445c-b812-2eca5775184a`
+- **SP deployment:** `/etc/gururmm-signing.env` on 172.16.3.30 (mode 0600, root)
+
+### Azure Trusted Signing — account
+
+- **Subscription:** Basic (`e507e953-2ce9-4887-ba96-9b654f7d3267`)
+- **Account:** `gururmm-signing` in rg `gururmm-signing-rg`, westus2
+- **Endpoint:** https://wus2.codesigning.azure.net/
+- **Cert profile:** `gururmm-public-trust`, Public Trust, subject `CN=Arizona Computer Guru LLC, O=Arizona Computer Guru LLC, L=Tucson, S=Arizona, C=US`
+- **EKU:** `1.3.6.1.4.1.311.97.842500947.753485959.130432357.333604967`
+- **Cert lifetime:** 3 days (by design; per-sign rotation)
+- **Timestamp URL:** http://timestamp.acs.microsoft.com (RFC3161 mode when using jsign)
+
+### GuruRMM admin
+
+- **Public API:** https://rmm-api.azcomputerguru.com
+- **Dashboard:** https://rmm.azcomputerguru.com
+- **Admin email / password:** `admin@azcomputerguru.com` / `GuruRMM2025`
+
+### GuruRMM server (172.16.3.30 / Ubuntu 22.04)
+
+- **SSH:** `guru` / `Gptf*77ttb123!@#-rmm` (vault: `infrastructure/gururmm-server.sops.yaml`)
+- **Sudo password:** same as SSH
+- **Postgres:** local, db `gururmm`, user `gururmm`, password `43617ebf7eb242e814ca9988cc4df5ad`
+- **JWT secret:** `ZNzGxghru2XUdBVlaf2G2L1YUBVcl5xH0lr/Gpf/QmE=`
+- **MariaDB (ClaudeTools):** db `claudetools`, user `claudetools`, password `CT_e8fcd5a3952030a79ed6debae6c954ed`
+
+### Business identity (for cert recovery / re-verification)
+
+- **Legal entity:** Arizona Computer Guru LLC
+- **D-U-N-S:** 00-566-1506 (005661506)
+- **EIN:** 20-5419777
+- **Address:** 7437 E 22nd St, Tucson, AZ 85710
+- **Phone:** +1 520 304 8300
+- **Identity Validation ID:** `03028768-f611-4904-aa58-c755020f436a`
+
+### Gitea
+
+- **URL:** https://git.azcomputerguru.com
+- **User:** azcomputerguru / `Gptf*77ttb123!@#-git`
+- **API token:** `9b1da4b79a38ef782268341d25a4b6880572063f`
+- **SSH:** `ssh://git@172.16.3.20:2222`
+
+## Infrastructure & Files Touched
+
+### On 172.16.3.30 (GuruRMM / build server)
+
+| Path | Purpose |
+|---|---|
+| `/etc/gururmm-signing.env` | SP credentials for jsign (mode 0600, root) |
+| `/opt/gururmm/sign-windows.sh` | jsign wrapper — OAuth + TRUSTEDSIGNING storetype |
+| `/opt/gururmm/build-agents.sh` | CI build script, now signs Windows binary in-place |
+| `/opt/gururmm/build-agents.sh.bak-pre-signing` | Backup of original build script |
+| `/opt/gururmm/webhook-handler.py` | Gitea webhook handler (listens 127.0.0.1:9000) — webhook not wired from Gitea side |
+| `/opt/jre/` | Eclipse Temurin JRE 21 (tarball install) |
+| `/usr/bin/jsign` | jsign 7.1 (via deb) |
+| `/home/guru/.dotnet/tools/wix` | WiX — doesn't work on Linux, leaving installed in case useful later |
+| `/etc/nginx/sites-available/gururmm` | Added `Cache-Control: public, max-age=300, must-revalidate` on `/downloads/` |
+| `/var/www/gururmm/downloads/gururmm-agent-windows-amd64-0.6.0.exe` | Signed v0.6.0 |
+| `/var/www/gururmm/downloads/gururmm-agent-windows-amd64-0.6.0.exe.bak-unsigned-20260416` | Unsigned rollback |
+| `/var/www/gururmm/downloads/gururmm-agent-windows-amd64-0.6.1.exe` | Signed v0.6.1 (sha256 `0b8acee8...`) |
+| `/var/www/gururmm/downloads/gururmm-agent-linux-amd64-0.6.1` | Linux v0.6.1 |
+| `/var/www/gururmm/downloads/gururmm-agent-windows-amd64-latest.exe → ...-0.6.1.exe` | Symlink |
+| `/var/www/gururmm/downloads/gururmm-agent-linux-amd64-latest → ...-0.6.1` | Symlink |
+
+### On user's Windows workstation
+
+| Path | Purpose |
+|---|---|
+| `C:\tools\trusted-signing\` | sign.ps1, metadata.json, dlib — from yesterday |
+| `C:\Users\guru\.dotnet\tools\wix.exe` | WiX 5.0.2 |
+| `C:\Program Files\dotnet\` | .NET SDK 8.0.420 (installed today via winget) |
+| `D:\work\gururmm\` | Fresh git clone of `azcomputerguru/gururmm` — used to push v0.6.1 changes |
+
+### In this repo (claudetools)
+
+| Path | Purpose |
+|---|---|
+| `projects/msp-tools/guru-rmm/signing-attestation/signing-config/sign-windows.sh` | jsign wrapper (pulled from server) |
+| `projects/msp-tools/guru-rmm/signing-attestation/signing-config/build-agents.sh` | Build script (pulled from server) |
+| `projects/msp-tools/guru-rmm/signing-attestation/signing-config/binaries/` | Local copies of signed test binaries |
+| `projects/msp-tools/guru-rmm/installer/gururmm.wxs` | WiX installer definition |
+| `projects/msp-tools/guru-rmm/installer/build-msi.ps1` | Build + sign wrapper for MSI |
+| `projects/msp-tools/guru-rmm/installer/README.md` | Installer docs |
+| `projects/msp-tools/guru-rmm/installer/.gitignore` | Excludes *.msi, *.wixpdb, src/*.exe |
+
+### In vault repo
+
+- `services/azure-trusted-signing.sops.yaml` — updated with SP credentials, IV passed status, cert profile details, test sign results. Fully encrypted with age.
+
+## Important Commands (reference)
+
+### Sign a Windows binary on the build server
+
+```bash
+sudo /opt/gururmm/sign-windows.sh /path/to/binary.exe "Optional description"
+```
+
+### Trigger a build manually (when webhook doesn't fire)
+
+```bash
+ssh guru@172.16.3.30
+sudo /opt/gururmm/build-agents.sh
+```
+
+### Build + sign an MSI on Windows
+
+```powershell
+cd D:\claudetools\projects\msp-tools\guru-rmm\installer
+.\build-msi.ps1 -Version 0.6.1
+```
+
+### Verify a signed binary
+
+```powershell
+& 'C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x64\signtool.exe' verify /pa /v <file>
+```
+
+### Rotate the SP credential
+
+```bash
+az ad sp credential reset --id 516d0bdc-5416-4d02-8521-b70e2bb26d29
+# Update /etc/gururmm-signing.env with the new secret
+# Update vault services/azure-trusted-signing.sops.yaml
+```
+
+## Problems Encountered + Resolutions
+
+| Problem | Resolution |
+|---|---|
+| `dotnet sign` tool crashes on Linux with `kernel32.dll.so not found` | Not actually cross-platform. Switched to jsign (Java, verified cross-platform). |
+| jsign Authenticode timestamp mode fails against ACS URL | Force `--tsmode RFC3161` |
+| Build server can't reach Ubuntu apt mirrors (IPv6 routing problem) | Installed Temurin JRE from tarball, jsign from GitHub deb release directly. Not ideal but works. Real fix: fix IPv6 routing on the network, not our problem today. |
+| VERSION came out empty in first build | `sed '\1'` backreference eaten through heredoc. Switched to `awk -F'"' '/^version/{print $2; exit}'` |
+| WiX 7 requires OSMF EULA | Stepped back to WiX 5.0.2 |
+| WiX officially Windows-only despite .NET tooling | Build MSI on Windows workstation; ship to Linux server for hosting |
+| Gitea webhook didn't trigger build on `git push` | Manually ran build-agents.sh; webhook config is an open item |
+| Cloudflare edge cached old unsigned v0.6.0 | Added nginx `Cache-Control` header for future builds; waited/manual purge for this one |
+| `Get-AuthenticodeSignature` returned empty from bash-spawned PowerShell | PS-from-bash quirk; signtool verify from direct invocation works |
+
+## Pending / Open Items
+
+1. **Configure Gitea webhook** on `azcomputerguru/gururmm` repo to POST to `http://172.16.3.30:9000/webhook/build` (or whatever the handler expects). Per the existing `/opt/gururmm/webhook-handler.py`, it listens on 127.0.0.1:9000 with an optional `X-Gitea-Signature` HMAC-SHA256 using `WEBHOOK_SECRET=gururmm-build-secret` (env var default). Need to either expose this externally (nginx proxy) or configure Gitea's outbound webhook to hit an internal path.
+
+2. **Provision Jupiter Windows VM** per spec above (Server 2022, 4 CPU, 8 GB, 80 GB). Awaiting user decision on hypervisor (Unraid QEMU vs Hyper-V).
+
+3. **Draft `setup-windows-build-worker.ps1`** once VM is up — installs .NET SDK, WiX 5, Rust + Windows target, VS Build Tools, OpenSSH Server, Gitea runner, Trusted Signing tooling.
+
+4. **MSI Phase 2** — service registration, site-code injection, agent install custom action. Best done on the Jupiter VM once up.
+
+5. **Agent fleet update to v0.6.1** — AD2 and DESKTOP-0O8A1RL still on v0.6.0 at end-of-day. Should auto-update on next poll; verify.
+
+6. **Cloudflare cache for the v0.6.0 swap** — may still be serving stale unsigned binary to some edge regions until ~2h TTL expires. Non-urgent but manual purge via CF dashboard would accelerate.
+
+7. **Carry-over from yesterday:**
+   - Tunnel status 403 bug (server/src/db/tunnel.rs)
+   - Drop `test_records_dedup_bak_20260415` on Dataforth DB after regressions verified
+   - Cleanup scratch `_*.js` scripts in `C:\Shares\testdatadb\database\` on AD2
+
+## Key Reference URLs
+
+- Public API: https://rmm-api.azcomputerguru.com
+- Download listing: https://rmm-api.azcomputerguru.com/downloads/
+- Azure Portal (subscription): https://portal.azure.com/#@/resource/subscriptions/e507e953-2ce9-4887-ba96-9b654f7d3267/resourceGroups/gururmm-signing-rg/providers/Microsoft.CodeSigning/codeSigningAccounts/gururmm-signing/overview
+- jsign docs: https://ebourg.github.io/jsign/
+- WiX v5 docs: https://wixtoolset.org/docs/
+
+## Commits This Session
+
+- claudetools `2937c29` — fix VERSION parsing in build-agents.sh (awk, not sed)
+- claudetools `148ac75` — installer MVP (WiX wxs, build script, README)
+- claudetools `fdd0bb0` — sign-windows.sh + updated build-agents.sh
+- gururmm `b5bc068` — agent v0.6.1 with tracing-appender file logging
+- vault `8826292` — SP credentials for build-signer
+- vault `1df2f83` — cert profile + first test sign details
+
+## Success Metrics
+
+- **First successful test sign on Linux build server:** 2026-04-16 14:39 UTC
+- **First end-to-end signed release via CI pipeline:** v0.6.1 at 2026-04-16 14:57 UTC
+- **First signed MSI:** 2026-04-16 15:15 UTC (gururmm-agent-0.6.1.msi, 1.16 MB)
+- **Full Microsoft cert chain validates** through signtool from Windows workstation for both .exe and .msi
+- **Billing impact:** Trusted Signing Basic ~$9.99/mo + per-signature fees (fractional cents each). SP creation, cert profile creation, jsign — all free.