---
name: GuruRMM technical reference — server, API, user_session, pipeline, agent sandbox
description: Operational reference for GuruRMM — server layout (SSH user, paths on 172.16.3.30), agent downloads dir + channel-tag rollout control, privileged server access via the server's OWN root RMM agent (no SSH needed) + plink fallback, API auth + command execution + polling, user_session context (WTS impersonation, when SYSTEM fails), build-pipeline vendoring at deploy/build-pipeline/ (auto-sync to /opt/gururmm), Linux agent systemd sandbox trap (ProtectSystem=strict makes fs/mount observations sandbox-local).
type: reference
---

Rules: [[feedback_gururmm]]. Project state + principles + pending setup: [[project_gururmm]].

---

## Server layout (172.16.3.30)

SSH user is **`guru`**, not `mike`. Home is `/home/guru/`. Other users with home dirs: `gitea-runner` only.

- **Repo:** `/home/guru/gururmm`
- **Dashboard build:** `cd /home/guru/gururmm/dashboard && npm run build`
- **Deploy:** `sudo cp -r dist/* /var/www/gururmm/dashboard/`
- **Other dirs under `/home/guru/`:** `guru-connect`, `guruconnect-server`, `backups`

---

## Privileged server access — downloads dir, channel tags, root agent (no SSH needed)

**Agent downloads dir: `/var/www/gururmm/downloads`** (NOT the code default `/var/www/downloads`; set via `DOWNLOADS_DIR` env on the running `gururmm-server` process — read it live with `cat /proc/$(pgrep -f gururmm-server)/environ | tr '\0' '\n' | grep DOWNLOADS_DIR`). Holds the per-os/arch agent binaries (`gururmm-agent-{os}-{arch}-{version}[.exe]`), the base enrollment MSI, `latest` symlinks, `.sha256`, and **`.channel` sidecars**.

**Channel-tag rollout control (this is how beta/stable is gated):** each binary has a `<binary>.channel` file containing `stable` or `beta`. `scanner.rs::get_latest_version`: **beta** agents get the absolute-latest binary regardless of tag; **stable** agents get only the latest `stable`-tagged binary (no sidecar = stable). So to soak a release beta-first: `echo beta > <binary>.channel` for the new version's binaries; to promote: `echo stable > ...`. The build pipeline's cleanup keeps only the current version, so once a new version is beta-tagged stable agents find NO newer stable binary and simply stay put. (Done 2026-06-01 to hold agent 0.6.51 / the Windows BSOD feature on beta — re-tagged the 4 `gururmm-agent-windows-*-0.6.51.exe.channel` files to beta. See [[feedback_gururmm_build_channel_default]].)

**The server (172.16.3.30) runs its OWN GuruRMM Linux agent, AS ROOT** — hostname `gururmm` (resolve the UUID live via `GET /api/agents`; it was `5e5a7ebc-95ea-40c8-b965-6ec15d63e157` on 2026-06-01, but UUIDs change on re-enroll — never hardcode). This means **privileged commands on the server (read AND write the downloads dir, re-tag channels, inspect process environ, etc.) run through `/rmm` shell on that agent — no SSH required.** Contrary to the sandbox section below, real-path read/write to `/var/www/gururmm/downloads` works fine via this agent (verified by re-tagging channels 2026-06-01) — the `ProtectSystem` sandbox bites on mount *observations* and writes to paths missing from `ReadWritePaths`, not this dir. When unsure if a path is writable via the agent, just `touch` a tempfile and check.

**SSH fallback from GURU-5070 (Windows):** `sshpass` is NOT installed here (the ix-server memory's sshpass note does not apply to GURU-5070). Use **`plink` / `pscp`** at `C:\Program Files\PuTTY\` with `-pw` and the vault creds (`guru@172.16.3.30`, password in `infrastructure/gururmm-server.sops.yaml` → `credentials.password`; sudo password = SSH password). Prefer the root-agent path above for one-off server ops.

---

## API — execute a script on any agent

**Base:** `http://172.16.3.30:3001` (reachable from HOWARD-HOME and similar dev machines via Tailscale).

**Auth:** `infrastructure/gururmm-server.sops.yaml` → `credentials.gururmm-api.admin-email` + `admin-password`. Login returns a JWT valid for ~24h (86400s from iat).

### Flow

```bash
VAULT="$PWD/.claude/scripts/vault.sh"
EMAIL=$(bash "$VAULT" get-field infrastructure/gururmm-server.sops.yaml credentials.gururmm-api.admin-email)
PASS=$(bash  "$VAULT" get-field infrastructure/gururmm-server.sops.yaml credentials.gururmm-api.admin-password)

JWT=$(curl -s -X POST http://172.16.3.30:3001/api/auth/login \
  -H "Content-Type: application/json" \
  -d "{\"email\":\"$EMAIL\",\"password\":\"$PASS\"}" \
  | python -c "import json,sys; print(json.load(sys.stdin)['token'])")

# Find agent
curl -s http://172.16.3.30:3001/api/agents -H "Authorization: Bearer $JWT"

# Submit (json-encode to preserve quotes/newlines)
AGENT="<agent-uuid>"
PAYLOAD=$(python -c "
import json
with open('path/to/script.ps1','r',encoding='utf-8') as f: s=f.read()
print(json.dumps({'command_type':'powershell','command':s}))
")
RESP=$(curl -s -X POST http://172.16.3.30:3001/api/agents/$AGENT/command \
  -H "Authorization: Bearer $JWT" -H "Content-Type: application/json" -d "$PAYLOAD")
CMD_ID=$(echo "$RESP" | python -c "import json,sys; print(json.load(sys.stdin)['command_id'])")

# Poll
while true; do
  STATUS=$(curl -s http://172.16.3.30:3001/api/commands/$CMD_ID -H "Authorization: Bearer $JWT" \
    | python -c "import json,sys; print(json.load(sys.stdin)['status'])")
  [ "$STATUS" != "running" ] && break
  sleep 5
done

# Fetch result
curl -s http://172.16.3.30:3001/api/commands/$CMD_ID -H "Authorization: Bearer $JWT"
```

### Required fields & response

`POST /api/agents/:id/command` requires `command_type` (use `powershell` for Windows agents — the API accepts any string but Windows agent only runs powershell-compatible) and `command` (script text, JSON-encoded).

Response from `/api/commands/:cmd_id`:
```json
{
  "id": "uuid", "agent_id": "uuid", "command_type": "powershell",
  "command_text": "...", "status": "completed",   // running | completed | failed | timeout
  "exit_code": 0, "stdout": "...", "stderr": "...",
  "created_at": "ISO-8601", "started_at": "ISO-8601", "completed_at": "ISO-8601"
}
```

### When to use / not to use

**Use** for diagnostic checks on any enrolled agent, one-off remediation without ScreenConnect, anywhere you'd ask a user to paste a script.

**Don't** when the agent isn't enrolled (`GET /api/agents` first), for interactive sessions (no stdin), for scripts >1 MB (untested — keep modular).

**Notes:** `command_type: "powershell"` runs in SYSTEM context on Windows (agent runs as LocalSystem). Idempotent commands only — no rollback. If output is large, have the script write to a file on the agent and fetch via a separate command. Tunnel API (`/api/v1/tunnel/...`) is a planned interactive feature per `.claude/gururmm-tunnel-plan.md`, not deployed.

---

## `context=user_session` — run as the active logged-on user

`POST /api/agents/:id/command` accepts an optional **`context`** field (migration `041`):

- `"system"` (default) — Session 0 / SYSTEM. Original behavior.
- `"user_session"` — runs in the active logged-on user's desktop session via WTS token impersonation (`WTSQueryUserToken` + `DuplicateTokenEx` + `CreateProcessAsUserW`, in `agent/src/watchdog/wts.rs`). **Requires an active logged-on user on the endpoint.**

**Why it matters:** some Windows cmdlets fail as SYSTEM with "NonInteractive mode" / interactive-session errors and historically had to be done on-site. `user_session` runs them remotely instead. Verified 2026-05-27 on the Peaceful Spirit **BridgetteHome** L2TP VPN deploy: `Set-VpnConnection -L2tpPsk -AllUserConnection` — previously documented as "cannot be done remotely" — was set successfully via `user_session`, completing a VPN rollout entirely through RMM with no on-site visit.

**Elevation:** the WTS-impersonated token of a logged-on **admin** user comes back effectively elevated (`WindowsPrincipal.IsInRole(Administrator)=True`) — enough to write the all-user phonebook / HKLM. A **standard** logged-on user is NOT elevated, so admin-requiring commands still fail. Agent launches `powershell.exe -NonInteractive`; don't rely on real interactive prompts.

**Invoke:** `{"command_type":"powershell","command":"...","context":"user_session"}`. To dodge shell-quoting on multi-line scripts, base64-encode the script as UTF-16LE and send `powershell -NoProfile -NonInteractive -EncodedCommand <b64>` (`iconv` is absent in Git Bash — encode with `py`).

---

## Build-pipeline vendoring (`/opt/gururmm/` ⇄ repo `deploy/build-pipeline/`)

Pipeline runs at **`/opt/gururmm/`** on the gururmm server (root-owned, hand-maintained). The scripts had silently diverged from the repo (caused BUG-015 Windows build-gate gap). Reconciled 2026-06-01:

- **Source of truth:** scripts vendored in the gururmm repo at **`deploy/build-pipeline/`** — `build-{windows,linux,mac,agents,server,shared}.sh`, `sign-windows.sh`, `webhook-handler.py`, `README` (commit `2bf539e`).
- **Drift-stop (commit `24b5daf`):** `build-shared.sh` (runs first every build, after `git reset --hard origin/main`) `install -m 0755`-syncs the 6 build scripts from `deploy/build-pipeline/` → `/opt/gururmm/` each build. **Edit in repo + push to main → next build runs it.** No manual copy, no restart.
- **Two exceptions — manual `sudo cp` required** (can't self-overwrite mid-run):
  - `build-shared.sh` (the running puller).
  - `webhook-handler.py` (persistent HTTP server; also `sudo systemctl restart gururmm-webhook` to reload).
  They change rarely. See `deploy/build-pipeline/README.md`.
- Webhook still INVOKES the `/opt/gururmm` copies (not repo copies directly) — the sync keeps them current.
- Repo's older `scripts/webhook-handler.py` + `scripts/build-agents.sh` are a prior generation, superseded.
- `build-windows.sh`'s change-gate watches `agent/ installer/` (BUG-015 fix — installer-only `.wxs`/`.ico` changes now rebuild the MSI).

---

## Linux agent runs in a systemd sandbox — `findmnt` lies

The Linux agent (`gururmm-agent.service`) is hardened with **`ProtectSystem=strict`** → private mount namespace where `/` is read-only, only `ReadWritePaths=` entries are writable. **Every command dispatched through the agent runs inside that namespace** — so `findmnt /`, `touch`, `/proc/mounts` etc. report the **agent's sandboxed view, not the host's actual state**.

**Trap (hit 2026-06-01 on GURU-KALI):** I diagnosed "host root filesystem is read-only" because an RMM-dispatched `touch /var/lib/gururmm` returned EROFS (os error 30) and `findmnt /` showed `ro`. **The host root was rw the entire time** (SMART PASSED, ext4 clean, no kernel remount-ro). Real cause: the unit's `ReadWritePaths=` omitted `/var/lib/gururmm` → agent couldn't persist `/var/lib/gururmm/.device-id` → re-minted a `device_id` on each daily identity refresh → server (no `machine_uid` dedup) filed a new agent row each time (~11 ghosts).

**How to get host truth instead of sandbox view:**
- SSH to the host directly (commands run in the host namespace), OR
- Read the agent PID's namespace explicitly: `cat /proc/<agent_pid>/mountinfo` — the process-scoped `ro` on `/` is the tell that it's sandbox, not host. Compare against the host's `findmnt`.
- `errors=remount-ro` in a mount line is the stock default mount option — NOT evidence an error fired. Confirm an actual remount-ro with kernel `EXT4-fs error` logs + `dumpe2fs -h` error count.

**Fix pattern (additive):** drop-in `/etc/systemd/system/gururmm-agent.service.d/override.conf` with
```ini
[Service]
ReadWritePaths=/var/lib/gururmm
```
(systemd merges `ReadWritePaths` additively across drop-ins), then `daemon-reload` + `restart`.

**Better upstream fix:** `StateDirectory=gururmm` (handles dir creation + perms + RW bind in one directive).

**Fleet implication:** every systemd-installed GuruRMM Linux agent with this unit shape has the same latent bug until the installer is fixed. See filed todos (agent `ReadWritePaths` / `StateDirectory` + server `machine_uid` dedup).