azcomputerguru/claudetools
1
0
Fork 0
Code Issues 78 Pull Requests Actions Packages Projects Releases Wiki Activity

sync: auto-sync from DESKTOP-0O8A1RL at 2026-05-15 15:23:02

Browse Source 
Author: Mike Swanson
Machine: DESKTOP-0O8A1RL
Timestamp: 2026-05-15 15:23:02
This commit is contained in:
Mike Swanson
2026-05-15 15:23:05 -07:00
parent 6d6de33cb7
commit 31088cb8de
21 changed files with 1367 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

128
session-logs/2026-05-15-session.md
View File

@@ -521,3 +521,131 @@ curl http://localhost:3001/api/changelog/server/latest # 200
- **Key file**: `agent/src/transport/websocket.rs` — `last_incoming` at line ~279, `sleep_until` at line ~361
- **Key file**: `server/src/api/changelog.rs`
- **Key file**: `scripts/generate-changelog.sh`
---
## Update: 15:20 PT — Pluto SSH recovery, Defender removal, build pipeline repair, perf test
## User
- **User:** Mike Swanson (mike)
- **Machine:** DESKTOP-0O8A1RL
- **Role:** admin
- **Session span:** ~18:00 UTC – 22:20 UTC 2026-05-15 (continued from prior context window)
## Session Summary
The session opened with Pluto (172.16.3.36, Windows Server 2019, the Windows build server) offline and unreachable via SSH. Pluto had been unreachable since at least the prior session. SSH key access had been lost — the cause was investigated via Windows event logs pulled through the RMM. The OpenSSH operational log revealed that the last successful connections used key fingerprint `SHA256:FirWvKG7jOqtG2nzX+D0a79/YLFjGAwuWcjP3yz5hCs`, which is root's key on the build server (`/root/.ssh/id_ed25519`), not the guru user's key. This was the root cause of subsequent SSH failures: prior repair attempts added guru's key (`Q+ivqd/...`) instead of root's key. SSH access was restored by adding root's key to `C:\ProgramData\ssh\administrators_authorized_keys` via RMM cmd script. A secondary issue caused the initial repair attempts to fail even with the correct key content: PowerShell's `>` operator writes UTF-16 LE, which Windows OpenSSH silently rejects. The file must be written with explicit ASCII encoding via `[System.IO.File]::WriteAllText(..., [System.Text.Encoding]::ASCII)`. Once both the correct key and correct encoding were in place, SSH worked.
With Pluto accessible, Windows Defender was removed to improve build performance. `Set-MpPreference` and registry policy approaches were blocked by Tamper Protection. DISM failed due to wrong flag syntax for Server 2019. `Uninstall-WindowsFeature` fails over SSH due to a Windows console I/O buffer issue. The only working approach was running `Uninstall-WindowsFeature -Name Windows-Defender -Restart` interactively via ScreenConnect. Pluto rebooted, Defender was fully removed.
With Defender gone, the build pipeline was repaired end-to-end. Three separate issues prevented automatic builds from firing. First: Gitea 1.25.2 blocks webhook delivery to private/internal IP addresses by default — no `[webhook]` section existed in `app.ini`, so all push events were silently dropped. Fix: added `ALLOWED_HOST_LIST = *` to `app.ini` and restarted the Gitea container. Second: the webhook handler (`/opt/gururmm/webhook-handler.py`) used `subprocess.Popen` without ever calling `proc.wait()`, causing every completed build to leave a zombie sudo process. `os.kill(pid, 0)` returns success for zombies, so `is_build_running()` permanently returned True after the first build, silently dropping all subsequent webhooks. Fix: moved build execution to a daemon thread that calls `proc.wait()` and removes the lock file on completion. Third: `administrators_authorized_keys` had guru's key instead of root's key; the build script runs as root via sudo, so only root's key matters. Fix: added root's key via RMM alongside guru's key.
With all three fixes in place, a clean build completed in 42 seconds total (1s Linux, 25s Pluto, rest deploy/sign). The previous baseline with Defender enabled was 367 seconds — an 8.7x speedup. Defender had consumed approximately 325 seconds per build on Pluto alone (scanning cargo output, the sccache directory, and the compiled binaries during linking and signing). A Gitea webhook to the Pluto password (`Paper123!@#`) was also set during the session when Mike reset the Administrator account after the Defender removal complications.
## Key Decisions
- **ASCII encoding for authorized_keys**: PowerShell's `>` and `Out-File` default to UTF-16 LE. Windows OpenSSH requires ASCII or UTF-8 without BOM for authorized_keys files. Silently fails with no error message — looks like a permissions issue. Use `[System.IO.File]::WriteAllText` with `[System.Text.Encoding]::ASCII` exclusively.
- **Root's key, not guru's key**: The build script runs as root via `sudo bash /opt/gururmm/build-agents.sh`. SSH connections to Pluto use `/root/.ssh/id_ed25519`, not `/home/guru/.ssh/id_ed25519`. Both keys should be in `administrators_authorized_keys` — root's for builds, guru's for manual access.
- **Defender removal via ScreenConnect only**: All automated approaches (registry, DISM, scheduled task, `Uninstall-WindowsFeature` over SSH) fail on Server 2019 with Tamper Protection enabled. Interactive console is required. Not worth automating further.
- **Thread-based build dispatch in webhook handler**: Alternative was fixing `is_build_running()` to detect zombies via `/proc/<pid>/status`. Thread approach is cleaner: `proc.wait()` in the thread reaps the child and removes the lock atomically. Lock file is only present while the build is actively running.
- **No manual build runs**: Rule established (and saved to memory) — `build-agents.sh` must only be triggered via the Gitea webhook pipeline. Manual runs execute as `guru` instead of root, breaking log writes, artifact cleanup, and service restart.
## Problems Encountered
- **SSH key wrong user**: Added guru's key to Pluto instead of root's key. Build pipeline uses root. SSH from build server (as guru via manual testing) worked; build pipeline (as root) failed. Fixed by adding root's key via RMM.
- **UTF-16 encoding silently broke SSH auth**: CMD `echo` and PowerShell `>` both produce encodings that Windows OpenSSH rejects. No error in sshd logs — just falls through to password auth. Resolution: `[System.IO.File]::WriteAllText` with explicit ASCII encoding.
- **Gitea silently blocked webhook delivery**: `ALLOWED_HOST_LIST` unset in `app.ini` caused Gitea 1.25.2 to drop all push webhook deliveries to 172.16.3.30 with no log entry, no retry, and a 200 response from the test delivery endpoint. Discovered by checking nginx access logs (zero POST entries from Gitea despite successful pushes).
- **Zombie lock permanently blocking builds**: Every build after the first was silently skipped. `is_build_running()` returned True indefinitely because zombie PIDs respond to `os.kill(pid, 0)`. Discovered by checking lock file PID against `ps` — process showed `<defunct>`. Fixed by reaping child in a thread.
- **Gitea app.ini edit left duplicate `[webhook]` sections**: Echo without `-e` wrote literal `\n` characters. Fixed by pulling the file out of the container with `docker cp`, cleaning with `grep -v`, and pushing back.
- **`Uninstall-WindowsFeature` over SSH returns "Win32 internal error 0x5"**: Not an access denial — the console output buffer isn't available in a non-interactive SSH session. This specific cmdlet requires a real console. Cannot be automated over SSH.
## Configuration Changes
| Location | File/Resource | Change |
|---|---|---|
| Gitea container | `/data/gitea/conf/app.ini` | Added `[webhook]\nALLOWED_HOST_LIST = *` |
| Build server | `/opt/gururmm/webhook-handler.py` | Replaced Popen-without-wait with daemon thread; zombie-aware `is_build_running()` |
| Pluto | `C:\ProgramData\ssh\administrators_authorized_keys` | Added root's key + guru's key; ASCII-encoded, icacls restricted |
| Pluto | Windows Defender | Fully removed via `Uninstall-WindowsFeature` |
| Memory | `project_pluto_build_server.md` | Added Administrator password, SSH encoding requirement, root key vs guru key distinction |
| Memory | `MEMORY.md` | Added GuruRMM build rule entry |
| Memory | `feedback_gururmm_builds.md` | New: no manual builds, always use webhook pipeline |
## Credentials & Secrets
- **Pluto Administrator password**: `Paper123!@#` (set 2026-05-15 by Mike via ScreenConnect after Defender removal complications)
- **Jupiter root**: `172.16.3.20` / `root` / `Th1nk3r^99##` — from vault `infrastructure/jupiter-unraid-primary.sops.yaml`
- **Jupiter iDRAC**: `172.16.1.73` / `root` / `Window123!@#-idrac`
- **Gitea API token**: `9b1da4b79a38ef782268341d25a4b6880572063f` (azcomputerguru account) — from vault `services/gitea.sops.yaml`
- **RMM API**: `claude-api@azcomputerguru.com` / `ClaudeAPI2026!@#` — `http://localhost:3001/api`
## Infrastructure & Servers
- **Pluto**: `172.16.3.36`, Windows Server 2019, VM on Jupiter. SSH: `Administrator@172.16.3.36`. Build pipeline SSHes as root (uses `/root/.ssh/id_ed25519`). Manual access uses guru's key.
- **Jupiter**: `172.16.3.20`, Unraid primary. SSH: `root@172.16.3.20`. 125 GB RAM total, 92 GB used (80 GB VMs, ~8 GB Docker). 33 GB available.
- **Jupiter VMs**: Windows Server 2016 (32 GB), GuruRMM (16 GB), OwnCloud (16 GB), Claude-Builder (8 GB), Unifi (8 GB)
- **Jupiter notable Docker containers**: seafile-elasticsearch (1.86 GB / 2 GB limit — at capacity), app (1.39 GB), seafile (1.13 GB), gitea (852 MB)
- **Gitea**: Docker container on Jupiter, port 3000 (internal). External: `https://git.azcomputerguru.com` (via Cloudflare). Always use `http://172.16.3.20:3000` for API calls.
- **Build webhook**: `POST http://172.16.3.30/webhook/build` → nginx → `http://127.0.0.1:9000` → `gururmm-webhook.service` → `/opt/gururmm/webhook-handler.py`
## Commands & Outputs
```bash
# SSH to build server
ssh guru@172.16.3.30
# SSH hop to Pluto (from build server)
ssh -o StrictHostKeyChecking=no Administrator@172.16.3.36 hostname
# Jupiter RAM check
ssh root@172.16.3.20 "free -h"
# Mem: 125Gi total, 92Gi used, 808Mi free, 34Gi buff/cache, 33Gi available
# Gitea webhook test delivery
curl -s -X POST 'http://172.16.3.20:3000/api/v1/repos/azcomputerguru/gururmm/hooks/1/tests' \
-H 'Authorization: token 9b1da4b79a38ef782268341d25a4b6880572063f'
# Trigger build via empty commit (correct method)
ssh guru@172.16.3.30 "cd /home/guru/gururmm && git commit --allow-empty -m 'chore: trigger build' && git push"
# Restart Gitea after app.ini change
ssh root@172.16.3.20 "docker restart gitea"
# Check webhook handler zombie issue
cat /var/run/gururmm-build.lock # showed PID
ps -p <PID> # showed <defunct>
rm /var/run/gururmm-build.lock # cleared stale lock
```
Build performance results:
```
Baseline (Defender on, warm sccache): 367s total
Post-Defender (warm sccache): 42s total
Linux agent: 1s (fully cached)
Pluto: 25s (cargo + WiX + 4 binaries)
Deploy/sign: 16s
Speedup: 8.7x
```
## Pending / Incomplete Tasks
- **Pluto password not in vault**: `infrastructure/pluto-build-server.sops.yaml` doesn't exist yet. Password `Paper123!@#` is in memory only. Mike to add to vault.
- **BB-SERVER enrollment loop**: duplicate key `idx_agents_site_device` — pre-existing, unresolved.
- **Windows 0.6.21 not yet distributed**: Pluto builds produce 0.6.21 Windows artifacts on each run. After today's fixes, they should now deploy correctly on future pushes. Verify next build publishes Windows artifacts.
- **IMC1 Unicode escape sequence** in hardware inventory: unresolved.
- **Policy wiring plan** (`ticklish-questing-stallman.md`): Deferred.
- **Portal changelog page**: API exists, no dashboard UI.
- **seafile-elasticsearch at container memory limit** (1.86 GB / 2 GB): Monitor — may need limit raised.
- **macOS agent builds**: Not yet implemented.
- **pre-commit hook not executable** on build server: `hint: The '/home/guru/gururmm/scripts/hooks/pre-commit' hook was ignored because it's not set as executable` — emitted on every commit. Low priority but noisy.
## Reference Information
- **Build pipeline commits (gururmm)**: `7773f49`, `44fef95`, `6eed227`, `106fce9`, `3e9ef32`, `509f901` (all empty trigger commits from this session)
- **Pluto agent ID (RMM)**: `5316f56f-a1b3-4ac5-97ac-71ddf6a74d2e`
- **Root SSH key fingerprint** (build server, used by pipeline): `SHA256:FirWvKG7jOqtG2nzX+D0a79/YLFjGAwuWcjP3yz5hCs` — `/root/.ssh/id_ed25519.pub`
- **Guru SSH key fingerprint** (build server, manual access): `SHA256:Q+ivqd/K3eKMqvLdwlkvNWKxvp3NyLt17PcxDwtykFs` — `/home/guru/.ssh/id_ed25519.pub`
- **Webhook handler**: `/opt/gururmm/webhook-handler.py` — `gururmm-webhook.service`, port 9000
- **Build script**: `/opt/gururmm/build-agents.sh` (production, runs as root via webhook)
- **Gitea webhook ID**: 1, repo `azcomputerguru/gururmm`, event `push`, URL `http://172.16.3.30/webhook/build`
- **Gitea app.ini**: `/data/gitea/conf/app.ini` inside `gitea` Docker container on Jupiter