sync: auto-sync from GURU-5070 at 2026-06-15 09:41:53

Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-06-15 09:41:53

projects/graphifyy-eval/FINDINGS.md

@@ -0,0 +1,81 @@
+# Graphifyy vs GrepAI — findings (GURU-5070, 2026-06-15)
+
+## TL;DR
+On this machine, **Graphifyy does not clear the bar for day-to-day adoption.** Its code-graph
+is fast/free but largely redundant with GrepAI (already wired in); its one real differentiator
+(a graph over docs/PDFs) requires LLM semantic extraction that is **impractically slow on the
+apples-to-apples local-ollama config**, and making it usable would require a cloud LLM backend
+= ongoing API cost, which negates the local/free premise. Recommend **do not adopt**; keep GrepAI.
+
+## Arm A — GrepAI (baseline) [complete]
+10/10 answered, **18/20** rubric. Medians: ~3,025 ctx tokens/query, 3 calls, ~55s. Already
+indexed (no build step). Two notable retrieval misses: C1 (phrasing surfaced the old timeout
+reaper, not the comms-durability fix) and D2 (returned the SUPERSEDED kittle-design copy, missed
+the canonical June BEC report) — i.e. it trips on indexed stale duplicates. Cost: ~514k agent
+tokens for the 10-query arm.
+
+## Arm B — Graphifyy (local ollama) [BLOCKED at indexing]
+Setup done: `pip install graphifyy` (v0.8.39) + `openai` dep; GrepAI disabled per-machine
+(backed up). Backend = local ollama (apples-to-apples: GrepAI also uses ollama).
+
+Architecture confirmed:
+- **Code extraction = AST (tree-sitter), no LLM** — instant, free, local. Strong.
+- **Query/path/explain = local BFS over graph.json with a token budget** — cheap at query time.
+- **Doc/PDF/image extraction = generative LLM (JSON) per chunk via the chosen backend** — heavy.
+
+Indexing measurements (the blocker):
+- `msp-pricing` (2 code + 18 docs + 2 PDFs), `qwen3:8b`, --mode deep: **did NOT finish in 600s**;
+  graphify warned the 8B model is "too small for JSON instruction following" + VRAM/truncation.
+- `msp-pricing/docs` (3 markdown files), `codestral:22b`, --token-budget 4096: got through
+  **chunk 1 of 2 in 360s** and did not finish. ~minutes per chunk.
+- AST-only code extraction ran instantly in both runs.
+
+### Why local ollama is the wrong workload for Graphifyy (key insight)
+"Both use ollama" is true but the workloads differ fundamentally:
+- **GrepAI/ollama = embeddings** (nomic-embed-text): one fast forward pass per chunk. Cheap.
+- **Graphifyy/ollama = generative structured (JSON) extraction** per chunk: slow, needs a
+  large instruction-following model; small models fail JSON, large models are minutes/chunk.
+
+So the doc-graph that is Graphifyy's only edge over GrepAI is gated behind an indexing cost that
+is impractical locally on this hardware, and a cloud backend (gemini/claude/openai) would add
+real per-ingest API cost + break the local/free + ollama-parity premise.
+
+## Arm B addendum — Claude backend (cloud, breaks ollama-parity) [tested]
+To see if a capable cloud backend makes the doc-graph viable: re-ran the SAME 3 `msp-pricing/docs`
+files with `--backend claude` (key from vault `projects/gururmm/anthropic-api.sops.yaml`,
+`anthropic` pip dep added).
+- **Build: succeeded in 120s** (vs local-ollama DNF). 41 nodes, 68 edges, 9 communities.
+- **Cost reported by graphify: 7,813 in / 12,008 out tokens, ~$0.20 for 3 small docs** (~$0.068/doc).
+  Extrapolated: the doc slice (wiki + msp-pricing + kittle + dataforth + gc docs) is ~hundreds of
+  files = ~$10-$30 initial; the full repo's docs (wiki + hundreds of client session-logs/reports)
+  = ~$50-$200+; plus steady re-ingest as docs change (SHA256 cache skips unchanged, so steady-state
+  = changed/new docs only — a constant trickle in an active repo). Code stays free (AST).
+- **Query quality (the decisive finding):** `graphify query` is a local, free, 1s BFS over
+  graph.json. For "GPS pricing tiers and prices" it returned the **concept/relationship MAP**
+  (all tier + plan NODES, cross-doc concept links like "GPS Support Plans (Cross-Document
+  Concept)") in ~1,573 tokens — but **NOT the actual prices** ($19/$26/$39 absent; nodes carry
+  label + src file, not leaf values). To get the facts you still Read the source file. GrepAI
+  (Arm A D1) returned the file chunk WITH the prices and answered outright.
+
+### What this means
+- **Fact/content retrieval (the common day-to-day query):** GrepAI is more direct (returns
+  content). Graphifyy returns a map -> you still open the file. Extra hop.
+- **Structural/relationship retrieval (architecture, impact, cross-doc concept links):**
+  Graphifyy's genuine edge, and the cross-document concept synthesis is nice — but it's the rarer
+  query, and overlaps GrepAI's RPG for code.
+
+## Cost/benefit verdict (Mike's day-to-day)
+- Day-to-day is mostly MSP ops + bursts of dev. The retrieval that helps is code (dev bursts) +
+  docs/knowledge (client history). GrepAI already serves code well (18/20) and is zero-setup.
+- Graphifyy's code side ≈ redundant with GrepAI. Its doc side (the differentiator) can't be
+  cheaply/locally indexed here. Net marginal benefit is low; the standing index/maintenance cost
+  (and the second-system overhead) is real.
+- **Recommendation: do not adopt fleet-wide; do not replace GrepAI.** Revisit only if (a) a fast
+  local generative model + GPU make doc-graph indexing cheap, or (b) the doc/PDF knowledge-graph
+  becomes a must-have and a metered cloud-backend ingest budget is acceptable.
+
+## Reversal / cleanup
+- Re-enable GrepAI: restore `enabledMcpjsonServers` (backup at
+  `projects/graphifyy-eval/settings.local.json.bak`) — needs session restart.
+- Remove Graphifyy: `py -m pip uninstall -y graphifyy` (and `openai` if unwanted). Delete
+  `projects/graphifyy-eval/out/` scratch graphs.

projects/graphifyy-eval/PROTOCOL.md

@@ -0,0 +1,106 @@
+# Graphifyy vs GrepAI — evaluation protocol (GURU-5070)
+
+Goal: real, comparable data on whether **Graphifyy** beats the incumbent **GrepAI** for
+Mike's day-to-day in ClaudeTools, enough to make an adopt / skip / adopt-narrowly call.
+Decision hinges on token efficiency + retrieval quality, weighed against maintenance cost.
+
+## Tools under test
+- **GrepAI** — `D:\claudetools\grepai.exe mcp-serve`, exposed as `mcp__grepai__*` (semantic
+  search + RPG graph: explore / trace_callers / trace_callees / trace_graph). Repo-wide index
+  already built (`.grepai/`). Enabled per-machine via `enabledMcpjsonServers:["grepai"]` in
+  `.claude/settings.local.json`.
+- **Graphifyy** — `pip install graphifyy && graphify install`. Local graph (NetworkX +
+  tree-sitter + Leiden). CLI/skill: `graphify <path> [--mode deep|--update]`,
+  `graphify query "q"`, `graphify path "A" "B"`, `graphify explain "C"`. Docs/PDF/images
+  ingested via Claude API (token cost); code parsed locally.
+
+## Arms (run in separate sessions; MCP toggles need a restart)
+- **A — GrepAI** (baseline / "before"): grepai ON, Graphifyy not used. Run FIRST, this session.
+- **B — Graphifyy** ("after"): Graphifyy ON, grepai DISABLED (removed from
+  `enabledMcpjsonServers`). New session.
+- **C — Control** (optional): both off; only `grep`/`glob`/`Read`. Shows whether either graph
+  tool beats plain search.
+
+Same model for all arms. Each query answered in a FRESH sub-agent constrained to that arm's
+tools, to avoid cross-arm contamination. Scoring done against the rubric, blind to arm where
+feasible.
+
+## Fixed test corpus (both tools index the SAME slice)
+To keep it fair and bounded (not the whole repo + node_modules):
+- Code: `projects/msp-tools/guru-rmm/` (Rust server + agent + React dashboard)
+- Docs: `wiki/`, `projects/msp-pricing/`, `clients/kittle/`, `clients/dataforth/`
+- PDF: `projects/msp-pricing/marketing/The Arizona Business Owner's Guide to Choosing an MSP - Arizona Computer Guru.pdf`
+
+Note asymmetry: GrepAI's existing index is repo-wide (slight recall edge, more noise);
+Graphifyy indexes exactly this slice. All test queries are answerable from the slice.
+
+## Metrics (per query x arm)
+| Metric | How captured |
+|---|---|
+| `ctx_tokens` | chars of retrieved context the agent consumed / 4 (consistent approx) |
+| `tool_calls` | number of retrieval round-trips to reach the answer |
+| `latency_s` | wall-clock for the query |
+| `score` | 0 = wrong/missing, 1 = partial, 2 = complete & correct (vs rubric) |
+
+One-time / maintenance (measured once per tool):
+- `index_build_s` — full index of the test corpus (code-only, then code+docs)
+- `reindex_s` — incremental update after touching ONE file
+- `ingest_api_tokens` — Graphifyy's Claude-API tokens to ingest docs/PDF/images
+  (GrepAI: note its embedding model/cost; LLM-ingestion ≈ 0)
+
+## Test set (10 queries; code-heavy + docs-heavy, since docs is Graphifyy's claimed edge)
+Each has a rubric = key facts a correct answer MUST contain.
+
+CODE
+- C1: "In GuruRMM, how does the server avoid false-failing commands that were delivered but not
+  acked? Name the mechanism + migrations." Rubric: agent CommandAck on receipt + dedup; reaper
+  RE-DELIVERS un-acked instead of false-failing; migrations 058 acked_at / 059 delivery_attempts.
+- C2: "Trace where un-acked command re-delivery is handled in the RMM server and what calls it."
+  Rubric: the reaper fn + its caller path. (grepai trace_callers vs graphify path)
+- C3: "Where is GuruRMM agent self-update with rollback implemented and what guards it?"
+  Rubric: agent `updater/mod.rs` + watchdog.
+- C4: "What does GuruConnect SPEC-018 propose?" Rubric: session broker / capture worker as SYSTEM.
+
+DOCS / KNOWLEDGE (Graphifyy's claimed strength)
+- D1: "GPS pricing structure (tiers + prices)?" Rubric: Basic $19 / Pro $26 / Advanced $39 per
+  endpoint; support plans Essential/Standard/Premium/Priority.
+- D2: "Summarize the Kittle BEC/ACH-fraud incident and root cause." Rubric: Ken+marco+Accounting
+  compromised; fraudulent bank-change to City of Tucson + Marana ($130K+ prevented); IC3 filed;
+  root cause = April credential theft + incomplete remediation (password never reset, ~2mo).
+- D3: "Which ACG clients had M365 breach/credential incidents in 2026 and each root cause?"
+  Rubric (relationship query): Kittle (BEC), Dataforth (2026-03-27 phishing -> MFA), mvaninc
+  (unauthorized sign-in OKC). Partial credit per client.
+- D4: "List the 7 red flags of a bad MSP from the Buyers Guide." Rubric: the 7 from
+  MSP-Buyers-Guide-Content.md (unlimited-support, high-pressure sales, offshore-only, no
+  proactive monitoring, long lock-ins, one-size packages, no local presence). PDF/doc ingestion.
+- D5: "Canonical Kittle article path + what it superseded?" Rubric: clients/kittle.md canonical;
+  kittle-design.md superseded 2026-06-09.
+
+MIXED (code + docs)
+- M1: "How do new GuruRMM builds get promoted from beta to stable?" Rubric: builds tag beta;
+  promote via POST /api/updates/rollouts/:version/promote; build-server.sh auto-deploys.
+
+## Procedure
+1. (Arm A, now) For each query, spawn a sub-agent: tools = grepai + Read only; instruct it to
+   use ONLY grepai for retrieval, answer, and report (answer, total retrieved chars, # grepai
+   calls, elapsed). Log to results.csv with arm=A.
+2. Score each answer 0/1/2 vs rubric.
+3. Disable GrepAI (below), install + index Graphifyy, measure one-time costs.
+4. (Arm B, new session) Same queries, sub-agent tools = Bash(graphify) + Read; use ONLY
+   graphify for retrieval. Log arm=B. Score.
+5. (Arm C, optional) grep/glob/Read only. Log arm=C.
+6. Analyze: per-metric medians by arm; weight ctx_tokens + score (the day-to-day levers);
+   factor in index/maintenance cost and the doc-vs-code split.
+
+## Reversible environment changes (per-machine only)
+Disable GrepAI (edit `.claude/settings.local.json`, remove "grepai" from
+`enabledMcpjsonServers`; restart session). Re-enable = add it back. **Do NOT edit `.mcp.json`**
+(shared/fleet). Install Graphifyy: `py -m pip install graphifyy && graphify install`. Uninstall
+= `py -m pip uninstall graphifyy` + remove its skill. Snapshot of `settings.local.json` kept at
+`projects/graphifyy-eval/settings.local.json.bak` before any edit.
+
+## Open setup unknowns to resolve at install
+- Which API key/env var Graphifyy uses for doc/PDF/image ingestion (README didn't say; it bills
+  as "a Claude Code skill"). Confirm before indexing docs so ingest cost is attributable.
+- Whether `graphify query` itself spends LLM tokens to answer (vs returning raw graph context) —
+  affects per-query cost comparison; measure.

projects/graphifyy-eval/results.csv

@@ -0,0 +1,21 @@
+query,arm,ctx_tokens,grepai_or_graphify_calls,files_read,latency_s,score,notes
+C1,A,8500,6,0,75,1,"described OLD timeout reaper (mig 014/043) + interrupt-on-reconnect; MISSED the comms-durability CommandAck/dedup + re-deliver + migrations 058/059 (the actual fix). query-phrasing steered to wrong mechanism"
+C2,A,6000,3,2,76,2,"nailed it: fail_timed_out_commands (db/commands.rs:337) w/ acked_at/delivery_attempts gating + re-deliver via get_pending_commands (ws/mod.rs); caller=main.rs tokio task. (1 stray grep for a line number)"
+C3,A,3100,2,1,63,2,"updater/mod.rs AgentUpdater full flow + rollback watchdog + guards. complete"
+C4,A,1800,1,1,37,2,"SPEC-018 SYSTEM service host + session broker, capture workers as SYSTEM. complete"
+D1,A,2750,1,0,23,2,"GPS Basic $19/Pro $26/Adv $39 + 4 support plans + equip pack. complete, 1 call"
+D2,A,2412,1,0,33,1,"retrieved the SUPERSEDED clients/kittle-design/ April breach-check (Alexis/Ken inbox rules); MISSED the canonical June BEC/ACH-fraud event ($130K to City of Tucson/Marana prevented, IC3 filed). stale-duplicate retrieval"
+D3,A,12250,4,0,74,2,"open-ended relationship query; comprehensive + well-sourced (Valley Wide confirmed; Cascades/Bardach/Barbara blocked; Kittle unconfirmed). NOTE: my rubric was inaccurate - weak gold query"
+D4,A,2185,1,0,39,2,"all 7 red flags correct from MSP-Buyers-Guide-Content.md. (1 stray grep for titles)"
+D5,A,2950,3,0,56,2,"wiki/clients/kittle.md canonical, superseded kittle-design.md 2026-06-09. correct. (1 stray grep)"
+M1,A,7625,3,0,54,2,"beta-first + POST /api/updates/rollouts/:version/promote + .channel sidecars + dashboard promote. complete"
+C1,B,,,,,,
+C2,B,,,,,,
+C3,B,,,,,,
+C4,B,,,,,,
+D1,B,,,,,,
+D2,B,,,,,,
+D3,B,,,,,,
+D4,B,,,,,,
+D5,B,,,,,,
+M1,B,,,,,,