## User
- **User:** Howard Enos (howard)
- **Machine:** Howard-Home
- **Role:** tech

## Session Summary

Resumed the GuruRMM remote software-uninstall work (SPEC-030) from a prior window.
Reconstructed state from `session-logs/2026-06/2026-06-22-howard-gururmm-software-uninstall.md`:
the engine + knowledge-catalog branch `feat/engine-bcu-improvements` (head `0e8323b`) was
pushed but NOT merged, held for more validation. Per Howard's go-ahead ("as long as it doesn't
affect the non-beta rmm site we can push"), mapped the merge-to-main deploy model
(merge = build+deploy trigger; agent/server/dashboard land on BETA, prod dashboard promotion
is a separate deliberate `promote-dashboard.sh`). Confirmed the one nuance: a single shared
prod API serves both beta and prod dashboards, so merging deploys the server binary + runs
migrations against the prod DB — but core endpoints are untouched, migrations are additive,
and the prod UI is unaffected until promotion. Ran local pre-merge verification (server
`cargo check`, dashboard build, migration collision check — 062 new, 061 identical to main),
then merged PR #49 to main. Verified the deploy live: authenticated against the internal API
over Tailscale and confirmed `GET /api/software/knowledge` returned 200 `[]` (new binary live,
migration 062 applied).

Howard then reported a functional bug: running uninstall on several programs flashed a popup
window briefly, then nothing reported whether programs were removed; the per-row removable/unknown
labels worked but the action gave no feedback. Live forensics on the test agent showed ZERO
removal attempts recorded and ZERO knowledge entries despite a window flashing — proving the
engine ran partway but the result path recorded nothing. Howard redirected to a full audit +
polish of the uninstall function ("issues or missing functions not mapped like the unknown
flag"). Read all five layers (engine PS1, server `software.rs`, db `software_knowledge.rs` /
`software_removal.rs`, dashboard `SoftwareManager.tsx`, API `client.ts`) and produced a
seven-finding audit.

Root cause (Finding A): the engine emits its JSON results array ONLY at the end of the batch,
so any single target throwing OUTSIDE `Invoke-Uninstall`'s inner try (under StrictMode +
`EAP=Stop`) propagates to the main `catch` -> `exit 3` -> no JSON -> server records nothing and
the dashboard shows no result, even though programs removed before the throw already ran (the
flash). Implemented fixes across all three layers on branch `fix/software-uninstall-polish`,
verified (server cargo check, dashboard tsc + build, engine PS parse + dry-run), committed
(`bd6dd27`), and merged+deployed via PR #50. The server came up healthy (internal `/health` 200)
but end-to-end behavioral re-test is blocked: the test box DESKTOP-MS42HNC went offline ~47 min
before the deploy (VM powered off), so any uninstall returns `503 agent offline`.

## Key Decisions

- **Merged PR #49 (engine-bcu + catalog) under Howard's conditional go.** Confirmed the
  shared-server nuance explicitly before pushing; declined to promote the dashboard to prod.
- **Treated the no-feedback bug as an engine-robustness root cause, not a forensic hunt.**
  The 0-attempts/0-knowledge + window-flash evidence was conclusive; stopped fighting the
  GuruScan command-flood to retrieve the exact buried command record (cost > value).
- **Fixed at the engine first (always emit JSON), server second (tolerant parse).** Defense in
  depth: even if a future engine path exits non-zero with partial results, the server now
  records what came back instead of discarding the batch.
- **Per-device removal-history surfaced in the dashboard** rather than left as dead API methods
  — the recorded attempts + logs (migration 061) were invisible; this is the "function not
  mapped" Howard referenced.
- **Knowledge catalog cross-referenced onto inventory rows** ("the unknown flag"): a catalog-
  confirmed `requires_ui` program is now excluded from silent bulk-select even with an uninstall
  string; unknown/silent surfaced per-row.
- **Branched `fix/software-uninstall-polish` off the NEW main** (post-PR#49) rather than the old
  feature branch, to avoid re-introducing already-merged work.

## Problems Encountered

- **Test-box agent offline blocks end-to-end verification.** DESKTOP-MS42HNC last seen
  `2026-06-23T05:36:45Z` (~47 min before the PR#50 deploy) — VM powered off/asleep. The
  software list/uninstall endpoints return `503 agent offline` until it reconnects.
- **No clean remote way to confirm the exact binary-swap moment.** Cloudflare blocks dashboard
  fetches (JS challenge -> 403) even with a browser UA, and there's no password-SSH tool on
  Howard-Home (no plink/sshpass/Posh-SSH; only system OpenSSH) to tail `.30` build logs. Relied
  on: pipeline version-bump commit landing + internal `/health` 200 + auto-rollback safety net.
- **Gitea `vault.sh get-field credentials.api-token` mis-resolves to 4 chars** (recurring). Read
  the raw token by decrypting `D:/vault/services/gitea.sops.yaml` and grepping the `api-token`
  line. Token used via PowerShell `Invoke-RestMethod` against the public Gitea API (internal
  `172.16.3.20:3000` unreachable until Tailscale came up).
- **Git-Bash heredoc mangled `\\` to single backslash** in test-data JSON (even quoted `<<'EOF'`)
  -> invalid JSON -> engine `exit 3` ("Unrecognized escape sequence"). Rebuilt the test targets
  via PowerShell `ConvertTo-Json`. Logged as friction (errorlog, ref `feedback_tmp_path_windows`).
- **`Invoke-WebRequest` failed in NonInteractive mode** without `-UseBasicParsing`; switched to
  `Invoke-RestMethod` for API calls.
- **Large `/api/commands?limit=400` pull timed out** (2 min) — base64 engine bodies + GuruScan
  command flood make the list heavy. Abandoned the buried-record retrieval.

## Configuration Changes

Branch `fix/software-uninstall-polish` (guru-rmm submodule), commit `bd6dd27`, merged to main
via PR #50 (`e83b06c`):
- `agent/scripts/uninstall-engine.ps1` — per-target try/catch in the main loop; one bad target
  becomes a `failed` result row instead of aborting the batch; JSON array ALWAYS emitted.
- `server/src/api/software.rs` — tolerant result extraction (parse results from stdout even on
  non-zero exit; only fail when no array present); `-WindowStyle Hidden` on both dispatch
  wrappers (list + uninstall); factored out `extract_json_array()`.
- `server/src/db/software_knowledge.rs` — `#[allow(dead_code)]` on `get_by_name` (kept for
  future server-side row enrichment).
- `dashboard/src/components/SoftwareManager.tsx` — wired per-device removal-history panel
  (removalStatus/resolveRemoval) with logs + dismiss; cross-referenced fleet catalog onto
  inventory rows (requires_ui excluded from silent bulk-select; unknown/silent surfaced);
  StatusBadge explicit `unknown`/`dry-run`.

Already merged earlier this session (PR #49, `feat/engine-bcu-improvements` -> main `e83b06c`'s
parent line, merge `bce9e80`, version-bump `c399e70`): engine BCU improvements + knowledge
catalog (migration 062) + audit fixes — now live on the shared prod API.

Parent claudetools: `errorlog.md` (friction entry — heredoc JSON mangling).

## Credentials & Secrets

- No new credentials created. Reused (all already vaulted):
  - GuruRMM API admin: `claude-api@azcomputerguru.com` / `ClaudeAPI2026!@#` — vault
    `infrastructure/gururmm-server.sops.yaml` fields `credentials.gururmm-api.admin-email` /
    `admin-password`. Used for API auth during deploy verification.
  - Gitea API token: `9b1da4b79a38ef782268341d25a4b6880572063f` — vault
    `services/gitea.sops.yaml` field `credentials.api-token` (the `get-field` helper mis-resolves
    it; read via raw `sops -d`). Used for PR #49/#50 create+merge via the public Gitea API.
  - GuruRMM build host SSH: `guru` / `Paper123!@#-rmm` (sudo same) — vault
    `infrastructure/gururmm-server.sops.yaml`. Not used (no password-SSH tool available).

## Infrastructure & Servers

- GuruRMM API/server + build host: `172.16.3.30` (Ubuntu 22.04, user `guru`, repo
  `/home/guru/gururmm`). API on `:3001` (prod, shared by beta+prod dashboards). Internal
  `/health` -> 200. Postgres `:5432` localhost-only (not reachable over Tailscale). DB
  `gururmm` / user `gururmm` / pw `43617ebf7eb242e814ca9988cc4df5ad`.
- Public API: `https://rmm-api.azcomputerguru.com` (`/health` 200; SPA hosts 403 to non-browser).
- Beta dashboard `https://rmm-beta.azcomputerguru.com` (built from main). Prod dashboard
  `https://rmm.azcomputerguru.com` (promoted deliberately — UNCHANGED this session).
- Gitea internal `172.16.3.20:3000` (unreachable until Tailscale up); public
  `https://git.azcomputerguru.com` behind Cloudflare (Invoke-RestMethod works; curl blocked).
- Test box **DESKTOP-MS42HNC** (AZ Computer Guru / Howard-VM), agent id
  `0de89b88-b21d-4647-ab64-96157ba87cc5` — OFFLINE since `2026-06-23T05:36:45Z`.
- Internal network (172.16.3.x) reachable only over **Tailscale** from Howard-Home.

## Commands & Outputs

- Local pre-merge verify: `bash .claude/skills/gururmm-build/scripts/verify.sh server --check`
  (PASS), `... dashboard` (PASS), `... migrations` (062 next free, no collision).
- Deploy-live check: `POST /api/auth/login` then `GET /api/software/knowledge` -> 200 `[]`.
- Forensic: `GET /api/agents/<id>/software/removal-status` -> empty; `GET /api/software/knowledge`
  -> count 0 (proved nothing recorded). `GET /api/commands?agent_id=<id>&limit=60` -> all
  GuruScan monitoring (one per ~37s); uninstall commands scrolled past, 400-pull timed out.
- Engine dry-run smoke test (PowerShell ConvertTo-Json targets): exit 0, one row per target —
  `Fake MSI App` tier=msi, `Fake NSIS App` tier=silent, `No Metadata App` needs_interactive,
  `GuruRMM Agent` refused. PS parse: `[OK] engine parses clean`.
- PR create+merge via public Gitea API (`Invoke-RestMethod`): PR #49 -> merge `bce9e80`; PR #50
  -> merge `e83b06c`; pipeline bumps `c399e70` / `49ea0ef`.
- `list err 503` on software list post-deploy = agent offline (server returns 503 when the agent
  isn't WS-connected).

## Pending / Incomplete Tasks

- **Power on DESKTOP-MS42HNC** and let the agent reconnect — REQUIRED to verify the fix
  end-to-end. Then either Howard re-runs the uninstall from beta (expect per-program results +
  the new "Removal history - this device" panel to populate), or Claude runs a controlled
  single-program uninstall via the API (destructive — Howard names a throwaway).
- **Confirm the new server binary swapped** (couldn't pinpoint remotely) — implicitly proven once
  a post-deploy uninstall returns non-empty results (old broken engine returned nothing).
- **Post-deploy catalog checks** still open from the prior session: verify the catalog populates
  on a real uninstall + promote an unknown live on beta.
- **Prod dashboard promotion** NOT done (intentional). Promote deliberately later via
  `sudo /opt/gururmm/promote-dashboard.sh --confirm` when the beta UI is validated.
- Follow-ons unchanged: GuruConnect SPEC-019, rip-and-replace Tier 1.4, Tier 1.5, Linux/macOS.

## Reference Information

- guru-rmm main head: `e83b06c` (merge PR #50), pipeline bump `49ea0ef`. Fix commit `bd6dd27`
  on `fix/software-uninstall-polish`.
- Earlier merge: PR #49 `bce9e80`, bump `c399e70`. Pre-session main pin `1ec0256`.
- PRs: https://git.azcomputerguru.com/azcomputerguru/gururmm/pulls/49 and /50.
- Audit findings A-G (this session): A engine batch-abort (root cause, critical), B window flash,
  C server intolerant of non-zero exit, D removal-status not mapped to UI, E inventory rows
  ignore catalog ("unknown flag"), F StatusBadge missing unknown/dry-run, G `get_by_name` dead.
- Engine embed path: server `include_str!("../../../agent/scripts/uninstall-engine.ps1")`.
- Prior session log: `session-logs/2026-06/2026-06-22-howard-gururmm-software-uninstall.md`.
- Build model: `projects/msp-tools/guru-rmm/docs/BUILD.md`; pipeline
  `deploy/build-pipeline/README.md`. Promote: `/opt/gururmm/promote-dashboard.sh`.