sync: auto-sync from HOWARD-HOME at 2026-06-26 11:40:19

Author: Howard Enos Machine: HOWARD-HOME Timestamp: 2026-06-26 11:40:19
2026-06-26 11:40:52 -07:00
parent 0cf843e13d
commit 5bace24371
8 changed files with 473 additions and 0 deletions
--- a/clients/rednour/session-logs/2026-06/2026-06-26-howard-nick-mac-rmm-rootcause.md
+++ b/clients/rednour/session-logs/2026-06/2026-06-26-howard-nick-mac-rmm-rootcause.md
@@ -0,0 +1,187 @@
+## User
+- **User:** Howard Enos (howard)
+- **Machine:** Howard-Home
+- **Role:** tech
+
+## Session Summary
+
+Re-opened the deferred GuruRMM macOS-agent enrollment failure on Nick Pafford's Apple
+Silicon Mac at Rednour Law Offices (open item from the 2026-06-25 session). Howard is
+heading onsite and wanted the problem diagnosed and a fix pre-staged before arrival. The
+device was offline (present in ScreenConnect but not connected), so no live commands could
+be run against it — all diagnosis was done from the repo and the RMM server endpoints.
+
+Disproved the prior working hypothesis. The 2026-06-25 log, the wiki, and coord todo
+6f2d22be all recorded the cause as "served aarch64 binary is unsigned, so Apple Silicon
+SIGKILLs it (killed: 9)." Pulled the actual served binary from
+`https://rmm.azcomputerguru.com/install/GREEN-FALCON-7214/download/macos` (3,960,397 bytes,
+single-arch arm64) and parsed its Mach-O load commands directly: it carries an
+`LC_CODE_SIGNATURE` and a valid CodeDirectory with the adhoc flag set (linker-inserted
+ad-hoc signature, identifier `gururmm_agent-51a9f25b57c13649`). An ad-hoc-signed arm64
+binary satisfies Apple Silicon's AMFI and runs — so the unsigned/SIGKILL theory is wrong.
+All six linked dylibs are stock system frameworks (Security, CoreFoundation, libobjc,
+IOKit, libiconv, libSystem), ruling out a dyld "library not loaded" failure too. Howard
+recalled the actual error was a "file not found," which is consistent with this.
+
+Found the real root cause by reading the agent and server source. The server's enrollment
+endpoint (`server/src/api/enroll.rs`) types `EnrollRequest.site_id` as `uuid::Uuid` — it
+requires a UUID. The server-served macOS install script
+(`https://rmm.azcomputerguru.com/install/GREEN-FALCON-7214/macos`) writes the site **code**
+string `GREEN-FALCON-7214` into `/usr/local/etc/gururmm/site.plist` as `site_id`. The agent
+reads that and POSTs `site_id: "GREEN-FALCON-7214"` to `/api/enroll`, which fails UUID
+deserialization (HTTP 422) — enrollment retries forever and the agent never connects. The
+`.pkg` postinstall writes a real UUID into the same plist, so the two macOS installers
+disagree and the curl-script path is the broken one. The "file not found" Howard saw is the
+secondary symptom: `config.rs::default_config_path()` has no macOS branch, so a manual
+`gururmm-agent run` with no readable plist falls back to the Linux path
+`/etc/gururmm/config.toml`, which does not exist on macOS -> "Failed to read config file".
+
+Resolved the correct site UUID for Rednour's "Main" site via the RMM API
+(`c7f5787c-8e71-45b3-841f-fa52436f7d26`; client "Rednour Law Offices"
+`85f7cff4-d4db-48a8-b477-b8788122a361`, active). Note this differs from the `d008c7d4-...`
+UUID hardcoded in the repo's `.pkg` postinstall, which belongs to a different/test site and
+must not be used for Rednour. Built a self-contained onsite Terminal paste-block that
+installs the agent, writes `site.plist` with the UUID (not the code), writes the
+LaunchDaemon, reloads, and verifies. Delivered it to Howard's Discord DMs (split to fit the
+2000-char limit). Per Howard's instruction, did NOT update the wiki, coord todo, or message
+Mike with the proposed fix — all of that waits until the fix is verified onsite.
+
+## Key Decisions
+
+- **Verified the binary signature from the repo/server instead of trusting the recorded
+  hypothesis.** Parsing the Mach-O directly (load commands + CodeDirectory flags) overturned
+  the "unsigned" theory without needing the offline Mac.
+- **Use the site UUID `c7f5787c-...` from the live RMM API, not the `.pkg`'s hardcoded
+  `d008c7d4-...`.** The pkg UUID is for a different site; using it would enroll Nick's Mac
+  into the wrong site (or fail).
+- **Onsite workaround = run the official installer, then overwrite the one bad value
+  (code -> UUID) in `site.plist` and reload.** Uses the server's known-good plumbing for
+  everything except the buggy field. The Discord block writes the plist with the UUID from
+  the start so it is idempotent even on a clean machine.
+- **Hold all wiki/coord/Mike updates until verified onsite** (Howard's explicit call). The
+  records still say "unsigned binary / codesign fix"; correcting them before proof would
+  risk publishing a second unverified theory.
+- **Did not test enrollment from the workstation.** POSTing to `/api/enroll` would create a
+  junk `enrolled_agents` row for a fake hostname on production; the source already proves the
+  UUID requirement, so a live test was unnecessary and avoided.
+
+## Problems Encountered
+
+- **Prior root-cause hypothesis was wrong and was propagated to three places** (session log,
+  wiki known-issues, coord todo 6f2d22be). Disproved via direct Mach-O inspection. Records
+  intentionally left uncorrected pending onsite verification.
+- **First download URL guess 404'd.** `.../downloads/gururmm-agent-darwin-arm64` does not
+  exist; the real route is `/install/:site_code/download/macos` (read from
+  `server/src/main.rs`).
+- **Device offline in ScreenConnect** — no live diagnostics possible; all work done from
+  source + server endpoints, and the fix staged for Howard to run by hand onsite.
+- **Discord 2000-char limit** — the paste-block message was 2,091 chars and rejected (code
+  50035). Split the prose header off the code block (1,933 chars) and sent separately.
+
+## Configuration Changes
+
+No changes to client machines, the RMM server, or the repo were made this session (pure
+diagnosis + delivery). The agent on Nick's Mac was NOT modified — it was offline.
+
+Pending (to be applied onsite by Howard): write
+`/usr/local/etc/gururmm/site.plist` with `site_id =
+c7f5787c-8e71-45b3-841f-fa52436f7d26`, plus the LaunchDaemon, via the delivered paste-block.
+
+## Credentials & Secrets
+
+None created or discovered this session. The site enrollment code `GREEN-FALCON-7214` and
+site UUID `c7f5787c-8e71-45b3-841f-fa52436f7d26` are enrollment identifiers (not secrets);
+the agent obtains a per-agent key (`agk_...`) from the server only after a successful enroll.
+
+## Infrastructure & Servers
+
+- **Nick's Mac (target):** ScreenConnect machine name `DUXs-Mac-Studio`; Apple `Mac13,1`
+  Mac Studio, Apple M1 Max (arm64); macOS 26.5.1; serial F6QR2PN2R6. Confirm this is
+  actually Nick's box before enrolling (name suggests a "Dux" user).
+- **GuruRMM server:** `http://172.16.3.30:3001` (internal API used for site lookup);
+  public install/download host `https://rmm.azcomputerguru.com`; agent enroll API
+  `https://rmm-api.azcomputerguru.com/api/enroll`.
+- **Rednour "Main" site:** UUID `c7f5787c-8e71-45b3-841f-fa52436f7d26`, code
+  `GREEN-FALCON-7214`, active. Client "Rednour Law Offices"
+  `85f7cff4-d4db-48a8-b477-b8788122a361`.
+- **macOS agent install paths:** binary `/usr/local/bin/gururmm-agent`; plist
+  `/usr/local/etc/gururmm/site.plist`; LaunchDaemon
+  `/Library/LaunchDaemons/com.azcomputerguru.gururmm-agent.plist` (label
+  `com.azcomputerguru.gururmm-agent`, runs `gururmm-agent run`); log
+  `/usr/local/var/log/gururmm-agent.log`.
+
+## Commands & Outputs
+
+```bash
+# Served arm64 binary IS signed (overturns "unsigned" theory)
+curl -fsSL https://rmm.azcomputerguru.com/install/GREEN-FALCON-7214/download/macos -o agent
+# Mach-O parse: magic feedfacf (64-bit arm64), LC_CODE_SIGNATURE present,
+# CodeDirectory flags 0x20002 (adhoc set), identifier gururmm_agent-51a9f25b57c13649,
+# no CMS blob -> linker ad-hoc signature (runs on Apple Silicon).
+# Linked dylibs: Security, CoreFoundation, libobjc.A, IOKit, libiconv.2, libSystem.B (all stock).
+
+# Root cause in source:
+#   server/src/api/enroll.rs:29   pub site_id: uuid::Uuid   <- requires a UUID
+#   install script writes <key>site_id</key><string>GREEN-FALCON-7214</string>  <- a CODE
+#   => POST /api/enroll 422s, agent retries forever.
+#   agent/src/config.rs default_config_path() has no macOS branch -> /etc/gururmm/config.toml
+#   fallback => "Failed to read config file" (the observed "file not found").
+
+# Resolve correct Rednour Main site UUID
+eval "$(bash .claude/scripts/rmm-auth.sh)"   # RMM=http://172.16.3.30:3001
+curl -fsS -H "Authorization: Bearer $TOKEN" "$RMM/api/sites/c7f5787c-8e71-45b3-841f-fa52436f7d26"
+#   site: Main | client_id: 85f7cff4-... (Rednour Law Offices) | active: True
+```
+
+Onsite fix (delivered to Howard's Discord) — run in Terminal on the Mac:
+```bash
+sudo bash -c '
+UUID="c7f5787c-8e71-45b3-841f-fa52436f7d26"
+BIN="/usr/local/bin/gururmm-agent"; CFG="/usr/local/etc/gururmm/site.plist"
+PLIST="/Library/LaunchDaemons/com.azcomputerguru.gururmm-agent.plist"
+URL="https://rmm.azcomputerguru.com/install/GREEN-FALCON-7214/download/macos"
+mkdir -p /usr/local/bin /usr/local/etc/gururmm /usr/local/var/log
+curl -fsSL "$URL" -o "$BIN" && chmod +x "$BIN"
+printf "%s\n" "<?xml version=\"1.0\"?><plist version=\"1.0\"><dict><key>site_id</key><string>$UUID</string></dict></plist>" > "$CFG"; chmod 600 "$CFG"
+# (LaunchDaemon written per macos-install.sh), then:
+launchctl bootout system "$PLIST" 2>/dev/null || true
+launchctl bootstrap system "$PLIST"; launchctl enable system/com.azcomputerguru.gururmm-agent
+sleep 6; tail -30 /usr/local/var/log/gururmm-agent.log; plutil -p "$CFG"
+'
+# (full canonical block, with the real plist XML + LaunchDaemon, is in Howard's Discord DM)
+```
+
+## Pending / Incomplete Tasks
+
+- **Verify the fix onsite** (Howard, next): run the delivered paste-block on Nick's Mac.
+  Success = log shows "Enrollment complete - agent key persisted" + WS connect; `plutil -p`
+  shows an `agent_key`; agent appears in dashboard under Rednour Law Offices -> Main. Watch
+  for `Killed: 9` (would mean signing theory is wrong) or `Site not found`/422 (UUID typo).
+- **After verification — correct the records** (currently HELD): update wiki
+  `clients/rednour.md` (remove the unsigned/codesign known-issue, replace with the
+  code-vs-UUID enroll bug), close/annotate coord todo 6f2d22be, and file the bug for Mike.
+- **The actual code fix (for Mike):** either the macOS install script should stamp the site
+  UUID into `site.plist` (like the `.pkg` postinstall), or `/api/enroll` should accept a site
+  code (the legacy `register_legacy` handler already resolves via `get_site_by_code`).
+  Secondary: add a macOS branch to `default_config_path()` and finish the stubbed macOS
+  `install` verb (`main.rs` returns "macOS launchd service installation is not yet
+  implemented"). Until patched, every macOS enrollment needs the manual code->UUID step.
+- **Confirm `DUXs-Mac-Studio` is Nick's machine** before enrolling.
+- Carryover from 2026-06-25 (unchanged): Syncro #32343 billing (0.5h onsite), auto-reconnect
+  of the Documents mount in Nick's Login Items, return visit for phone + printer.
+
+## Reference Information
+
+- Coord todo: `6f2d22be-e653-48c8-9f9b-0155420b315d` (project gururmm) — still open, records
+  the now-disproven hypothesis; do not close until onsite verification.
+- Served macOS binary: `https://rmm.azcomputerguru.com/install/GREEN-FALCON-7214/download/macos`
+  (3,960,397 bytes, arm64, ad-hoc signed, id `gururmm_agent-51a9f25b57c13649`).
+- macOS install script: `https://rmm.azcomputerguru.com/install/GREEN-FALCON-7214/macos`.
+- Source files: `projects/msp-tools/guru-rmm/server/src/api/enroll.rs` (site_id: Uuid),
+  `agent/src/config.rs` (default_config_path, no macOS branch),
+  `agent/src/macos_storage.rs` (reads `/usr/local/etc/gururmm/site.plist`),
+  `agent/src/main.rs` (resolve_windows_config macOS branch; macOS install stub),
+  `agent/scripts/install.sh`, `agent/pkg-scripts/postinstall` (hardcoded `d008c7d4-...`).
+- Prior session: `clients/rednour/session-logs/2026-06/2026-06-25-howard-nick-smb-share-and-mac-rmm.md`.
+- Wiki: `wiki/clients/rednour.md` (macOS-unsigned known issue — to be corrected post-verify).
+- RMM agent (REDNOURCARRIEVI): `8e4e2221-7e2a-4a6f-9eda-864568539961`.