diff --git a/errorlog.md b/errorlog.md index 0e0c2a53..6d5e8634 100644 --- a/errorlog.md +++ b/errorlog.md @@ -19,6 +19,8 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure · +2026-07-07 | Howard-Home | rmm/windows-edition-upgrade | [correction] hand-rolled changepk Home->Pro instead of following reference_windows_edition_upgrade_rmm; changepk does NOT auto-reboot (flips edition live + sets RebootPending) so the long offline was the flaky RMM agent, not a reboot in progress - must trigger Restart-Computer -Force explicitly [ctx: host=DALLAS ref=reference_windows_edition_upgrade_rmm] + 2026-07-06 | Howard-Home | ssh/gururmm-server | [friction] vault infrastructure/gururmm-server.sops.yaml lists username/password + 'Sudo password same as SSH', but 172.16.3.30 SSH is publickey-only (password auth disabled) and no fleet key authenticates guru/root/mike/howard from workstations. Watching build logs / running promote-dashboard.sh must be done on the box. Fix: add SSH pubkey to guru@.30 or correct the vault note. [ctx: ref=infrastructure/gururmm-server.sops.yaml host=172.16.3.30] 2026-07-06 | Howard-Home | gururmm-build/webhook | [friction] linux webhook build SKIPPED after merge: change-detector excludes agent/Cargo.toml, so a Cargo.toml-only agent change (BUG-024 rustls swap) never rebuilds; had to force build-linux.sh manually as root [ctx: commit=a73dcad ref=build-linux.sh:29] diff --git a/session-logs/2026-07/2026-07-06-howard-bug024-musl-deploy-scileppi-wifi.md b/session-logs/2026-07/2026-07-06-howard-bug024-musl-deploy-scileppi-wifi.md new file mode 100644 index 00000000..d69a4b73 --- /dev/null +++ b/session-logs/2026-07/2026-07-06-howard-bug024-musl-deploy-scileppi-wifi.md @@ -0,0 +1,69 @@ +## User +- **User:** Howard Enos (howard) +- **Machine:** Howard-Home +- **Role:** tech + +## Session Summary + +Started by diagnosing a stray `C:\Program Files\Git\bin\bash.exe` console window on the Howard-Home desktop. Traced it to the `GPS-RMM-AutoEnroll` scheduled task (runs `gps-rmm-autoenroll.sh` every 30 min), which was visible because its principal used `LogonType: Interactive`. Confirmed the running instance was mid-`sleep 150` (normal, after pushing 3 installers), not hung. Changed the principal to `S4U` (runs hidden in Session 0), verified via a test run, then — per Howard's follow-up — disabled the task entirely (manual enrollment going forward) and killed the leftover Session-0 bash processes. + +Built the structure for a fleet-wide client Wi-Fi inventory so techs can connect onsite without asking. Security-driven split: passwords go encrypted in the SOPS vault (`clients//wifi.sops.yaml`, `credentials._ssid/_password`), and a human-readable index (no passwords) lives at `wiki/reference/client-wifi.md` (registered under a new "Reference" section in `wiki/index.md`, exempt from `/wiki-compile`). Per Howard's choices via AskUserQuestion: structure-only for now (the `/wifi` importer is deferred), dedicated wiki doc. Added a `reference_client_wifi_inventory` memory documenting the convention. + +Deployed the BUG-024 fix (Linux agent glibc-floor crash-loop). Verified GURU-5070's `fix/linux-musl-glibc-floor` branch (rustls on non-Windows, `x86_64-unknown-linux-musl` static, objdump build gate), merged it to gururmm `main` (`a73dcad`). Discovered the webhook SKIPPED the Linux rebuild — the change-detector excluded `agent/Cargo.toml`, and the fix was Cargo.toml-only. Forced `build-linux.sh` as root on `.30`; the objdump gate PASSED and published a verified glibc-free static `0.6.79` to beta (Linux un-quarantined). Then fixed the detector gap itself (BUG-025, `fe5468e`) in both `build-linux.sh` and `build-windows.sh`, validated on real commits. + +Attempted to recover the last still-dead Linux agent, Scileppi's SL-SERVER (Synology RS2212+). Howard stood up OpenVPN into Scileppi's network; SL-SERVER (192.168.242.5) became reachable but SSH resets at the banner (sshd restricts to the LAN subnet; my VPN IP 10.137.42.119 is off-subnet), and the vault DSM/user creds all fail DSM login (400, likely stale since the 2025-12-29 migration). The Mac mini jump-host path proved unreliable (T-Mobile public IP, and ScreenConnect returns no command output). Escalated to Mike via Discord DM. + +Closed the session by reviewing the three bugs touched: marked BUG-024 and BUG-025 FIXED in `FEATURE_ROADMAP.md`, then dug into why BUG-024's crash-loop never self-recovered. Root-caused it: the unix rollback watchdog is spawned inside the agent's systemd cgroup (`setsid` changes session, not cgroup), so `KillMode=control-group` (default; unit sets none) SIGKILLs it during the restart it's meant to guard — and it's the only rollback path on unix. Filed as BUG-026 (OPEN/HIGH) and, per Howard, flagged it for Mike to fix (coord todo + Discord DM) rather than implementing it. + +## Key Decisions +- Disabled (not deleted) `GPS-RMM-AutoEnroll` so it can be re-run manually; used S4U before deciding to disable, proving the hidden-run path works if re-enabled. +- Wi-Fi passwords go in the vault only; the wiki holds a passwordless index. Placed the index under `wiki/reference/` specifically because `/wiki-compile` only touches clients/projects/systems slugs, so it won't be clobbered. +- Forced the Linux build manually on `.30` rather than waiting on the webhook, because the change-detector gap meant the webhook would never rebuild a Cargo.toml-only change. Rebuilt as `0.6.79` (same version GURU-5070 validated), relying on the objdump gate as the safety net. +- Fixed BUG-025 by counting meaningful `Cargo.toml` lines (excluding only the `[package] version =` line) instead of excluding the manifest wholesale — preserves the version-bump false-positive guard while catching real dep/feature changes. +- Did NOT implement the BUG-026 watchdog fix; flagged for Mike per Howard. The fix (systemd-run --scope) touches the self-update path and needs testing on a real systemd host. +- Left `0.6.79` on beta to soak; stable promotion is a deliberate separate step. + +## Problems Encountered +- **Webhook skipped the BUG-024 Linux build.** `build-linux.sh`/`build-windows.sh` excluded `agent/Cargo.(toml|lock)`; the fix was Cargo.toml-only → `AGENT_CHANGED=0` → skipped. Resolved by forcing the build (cleared `/opt/gururmm/last-built-commit-linux` so the skip guard's `-n "$LAST_SHA"` was false), then fixed the detector (BUG-025). +- **Repeated sync races + a MEMORY.md rebase conflict.** GURU-5070 was auto-syncing frequently; three push races and one line-ending conflict in `.claude/memory/MEMORY.md` (Mike's normalized version vs my one-line add). Resolved by `git checkout --ours` + re-adding the single line. Also caught two empty `.ocu.err`/`.ocu.json` scratch files that `git add -A` swept in; removed from tracking and added `.ocu.*` to `.gitignore`. +- **Scileppi SL-SERVER unreachable for remediation.** SSH banner-reset (source-subnet restriction), DSM API creds fail (400), Mac mini on a foreign network + no SC command-output retrieval. Not resolvable without Mike changing NAS/VPN access or providing current creds; escalated. +- **`/tmp` write blocked by hook.** Wrote a JSON dump to `/tmp/mini.json`; `block-tmp-path.sh` correctly rejected it (Windows path mismatch rule). Switched to the scratchpad. + +## Configuration Changes +- `wiki/reference/client-wifi.md` — created (Wi-Fi inventory doc, no passwords). +- `wiki/index.md` — added "Reference" section + row. +- `.claude/memory/reference_client_wifi_inventory.md` — created; `.claude/memory/MEMORY.md` — index line added (survived a rebase conflict). +- `.gitignore` (claudetools) — added `.ocu.*`. +- Scheduled task `GPS-RMM-AutoEnroll` (Howard-Home) — principal set to S4U, then task Disabled. +- guru-rmm `deploy/build-pipeline/build-linux.sh` + `build-windows.sh` — BUG-025 detector fix. +- guru-rmm `docs/FEATURE_ROADMAP.md` — BUG-024 → FIXED, BUG-025 filed FIXED, BUG-026 filed OPEN. +- Server `.30`: `/opt/gururmm/last-built-commit-linux` cleared then repopulated; `/var/www/gururmm/downloads/gururmm-agent-linux-amd64-0.6.79` (+ .channel beta + .sha256) published; `/tmp/musl-build` worktree removed; `fix/linux-musl-glibc-floor` remote branch deleted. + +## Credentials & Secrets +- SL-SERVER (Scileppi Synology RS2212+) vault entry `clients/scileppi-law/rs2212.sops.yaml`: `credentials.username: sysadmin`, `credentials.password: Gptf*77ttb123!@#-sl-server` (the `\!` in the vault is shell-escape notation), users chris/andrew/sylvia/rose = `Scileppi2025!`. NOTE: **all of these FAIL DSM login (400)** — likely stale since the 2025-12-29 migration. No SSH private key is stored in the entry despite the "SSH key auth configured" note. Flag for update once working access is regained. +- gururmm server SSH: `infrastructure/gururmm-server-physical.sops.yaml`, `credentials.username: guru`, `credentials.ssh-private-key` (ed25519), sysadmin password `credentials.password`. Reaches `guru@172.16.3.30` (hostname gururmm). Key extracted to scratchpad during the session and deleted after use. + +## Infrastructure & Servers +- GuruRMM build/deploy server: `172.16.3.30` (hostname `gururmm`, Ubuntu 26.04, guru user, SSH key auth). Downloads dir `/var/www/gururmm/downloads`, quarantine `/var/www/gururmm/quarantine`, linux build log `/var/log/gururmm-build-linux.log`, last-built marker `/opt/gururmm/last-built-commit-linux`. +- Scileppi: SL-SERVER Synology RS2212+ at `192.168.242.5` (LAN 192.168.242.0/24); DSM API on :5000/:5001 reachable but auth fails; SSH :22 TCP-connects but resets at banner (subnet-restricted). Vault `host` field says 172.16.1.59 (unreachable — stale/mgmt IP). OpenVPN client IP this session: `10.137.42.119`. +- Mac mini "Mac-mini-2" (Sylvia, Apple M4) — ScreenConnect session `b9fb0065-a00e-08fe-bdbc-5d9d93c17245`, Company "The Law Offices of Chris Scileppi"; online but public IP `172.59.206.208` (T-Mobile, not Cox), so possibly off the office LAN. + +## Commands & Outputs +- `Set-ScheduledTask -TaskName 'GPS-RMM-AutoEnroll' -Principal (New-ScheduledTaskPrincipal -UserId 'Howard' -LogonType S4U -RunLevel Limited)` then `Disable-ScheduledTask`. +- Force build: `ssh guru@172.16.3.30 "sudo -S bash -c 'rm -f /opt/gururmm/last-built-commit-linux && /home/guru/gururmm/deploy/build-pipeline/build-linux.sh'"` → `BUG-024 gate PASSED — static musl binary, no glibc floor` → `Linux build complete: v0.6.79`. +- Verify static: `objdump -T | grep -c GLIBC_` = 0; `readelf -l | grep -c INTERP` = 0; `ldd` = "statically linked"; `file` = "static-pie linked". +- Detector validation: BUG-024 change `2471946` → 25 meaningful Cargo.toml lines (build); version bump `c81eeb5` → 0 (skip). +- Scileppi SSH: `kex_exchange_identification: read: Connection reset` (system ssh + paramiko), from first attempt. DSM login: all accounts `{"error":{"code":400}}`. + +## Pending / Incomplete Tasks +- **Scileppi SL-SERVER recovery (BUG-024 straggler)** — BLOCKED on access. With Mike (Discord DM `1523821940047876146`): needs VPN subnet `10.137.42.0/24` whitelisted for SSH on the NAS (or VPN NAT to LAN), OR current DSM creds, OR the authorized SSH key location. Vault creds `clients/scileppi-law/rs2212` are stale and need updating once in. +- **BUG-026** — flagged for Mike (coord todo `f7382334`, Discord DM `1523834569760178259`). Fix: spawn watchdog via `systemd-run --scope --collect` + bounded retry loop + verify agent stays up. +- **`0.6.79` stable promotion** — left on beta to soak; promote deliberately later (`/opt/gururmm/promote-*` or the pipeline). +- **`/wifi` importer** — deferred; build once a few real networks are captured. Schema in `wiki/reference/client-wifi.md`. + +## Reference Information +- gururmm commits: `a73dcad` (BUG-024 merge), `fe5468e` (BUG-025 detector fix), `c248214` (roadmap 024/025), `66152da` (roadmap 026). Branch `fix/linux-musl-glibc-floor` deleted post-merge. +- claudetools commits: `c9ea7e8` (wifi structure + .ocu ignore), later syncs `0b11a40`. +- Coord: 4 BUG-024 messages marked read (`cf35bf16`, `20daedf9`, `e2b01835`, `9659c5fc`); sent mike `5f8b33f7` (BUG-024 deploy) + `8c45b4dd` (detector fixed); coord todo `f7382334` (BUG-026, mike/gururmm). +- Discord DMs to mike: `1523821940047876146` (Scileppi access), `1523834569760178259` (BUG-026). +- Vault also synced an unrelated incoming entry `clients/altschuler/gururmm-site-main.sops.yaml`.