claudetools/projects/graphifyy-eval/PROTOCOL.md

Graphifyy vs GrepAI — evaluation protocol (GURU-5070)

Goal: real, comparable data on whether Graphifyy beats the incumbent GrepAI for Mike's day-to-day in ClaudeTools, enough to make an adopt / skip / adopt-narrowly call. Decision hinges on token efficiency + retrieval quality, weighed against maintenance cost.

Tools under test

• GrepAI — D:\claudetools\grepai.exe mcp-serve, exposed as mcp__grepai__* (semantic search + RPG graph: explore / trace_callers / trace_callees / trace_graph). Repo-wide index already built (.grepai/). Enabled per-machine via enabledMcpjsonServers:["grepai"] in .claude/settings.local.json.
• Graphifyy — pip install graphifyy && graphify install. Local graph (NetworkX + tree-sitter + Leiden). CLI/skill: graphify <path> [--mode deep|--update], graphify query "q", graphify path "A" "B", graphify explain "C". Docs/PDF/images ingested via Claude API (token cost); code parsed locally.

Arms (run in separate sessions; MCP toggles need a restart)

• A — GrepAI (baseline / "before"): grepai ON, Graphifyy not used. Run FIRST, this session.
• B — Graphifyy ("after"): Graphifyy ON, grepai DISABLED (removed from enabledMcpjsonServers). New session.
• C — Control (optional): both off; only grep/glob/Read. Shows whether either graph tool beats plain search.

Same model for all arms. Each query answered in a FRESH sub-agent constrained to that arm's tools, to avoid cross-arm contamination. Scoring done against the rubric, blind to arm where feasible.

Fixed test corpus (both tools index the SAME slice)

To keep it fair and bounded (not the whole repo + node_modules):

• Code: projects/msp-tools/guru-rmm/ (Rust server + agent + React dashboard)
• Docs: wiki/, projects/msp-pricing/, clients/kittle/, clients/dataforth/
• PDF: projects/msp-pricing/marketing/The Arizona Business Owner's Guide to Choosing an MSP - Arizona Computer Guru.pdf

Note asymmetry: GrepAI's existing index is repo-wide (slight recall edge, more noise); Graphifyy indexes exactly this slice. All test queries are answerable from the slice.

Metrics (per query x arm)

Metric	How captured
ctx_tokens	chars of retrieved context the agent consumed / 4 (consistent approx)
tool_calls	number of retrieval round-trips to reach the answer
latency_s	wall-clock for the query
score	0 = wrong/missing, 1 = partial, 2 = complete & correct (vs rubric)

One-time / maintenance (measured once per tool):

• index_build_s — full index of the test corpus (code-only, then code+docs)
• reindex_s — incremental update after touching ONE file
• ingest_api_tokens — Graphifyy's Claude-API tokens to ingest docs/PDF/images (GrepAI: note its embedding model/cost; LLM-ingestion ≈ 0)

Test set (10 queries; code-heavy + docs-heavy, since docs is Graphifyy's claimed edge)

Each has a rubric = key facts a correct answer MUST contain.

CODE

• C1: "In GuruRMM, how does the server avoid false-failing commands that were delivered but not acked? Name the mechanism + migrations." Rubric: agent CommandAck on receipt + dedup; reaper RE-DELIVERS un-acked instead of false-failing; migrations 058 acked_at / 059 delivery_attempts.
• C2: "Trace where un-acked command re-delivery is handled in the RMM server and what calls it." Rubric: the reaper fn + its caller path. (grepai trace_callers vs graphify path)
• C3: "Where is GuruRMM agent self-update with rollback implemented and what guards it?" Rubric: agent updater/mod.rs + watchdog.
• C4: "What does GuruConnect SPEC-018 propose?" Rubric: session broker / capture worker as SYSTEM.

DOCS / KNOWLEDGE (Graphifyy's claimed strength)

• D1: "GPS pricing structure (tiers + prices)?" Rubric: Basic $19 / Pro $26 / Advanced $39 per endpoint; support plans Essential/Standard/Premium/Priority.
• D2: "Summarize the Kittle BEC/ACH-fraud incident and root cause." Rubric: Ken+marco+Accounting compromised; fraudulent bank-change to City of Tucson + Marana ($130K+ prevented); IC3 filed; root cause = April credential theft + incomplete remediation (password never reset, ~2mo).
• D3: "Which ACG clients had M365 breach/credential incidents in 2026 and each root cause?" Rubric (relationship query): Kittle (BEC), Dataforth (2026-03-27 phishing -> MFA), mvaninc (unauthorized sign-in OKC). Partial credit per client.
• D4: "List the 7 red flags of a bad MSP from the Buyers Guide." Rubric: the 7 from MSP-Buyers-Guide-Content.md (unlimited-support, high-pressure sales, offshore-only, no proactive monitoring, long lock-ins, one-size packages, no local presence). PDF/doc ingestion.
• D5: "Canonical Kittle article path + what it superseded?" Rubric: clients/kittle.md canonical; kittle-design.md superseded 2026-06-09.

MIXED (code + docs)

• M1: "How do new GuruRMM builds get promoted from beta to stable?" Rubric: builds tag beta; promote via POST /api/updates/rollouts/:version/promote; build-server.sh auto-deploys.

Procedure

1. (Arm A, now) For each query, spawn a sub-agent: tools = grepai + Read only; instruct it to use ONLY grepai for retrieval, answer, and report (answer, total retrieved chars, # grepai calls, elapsed). Log to results.csv with arm=A.
2. Score each answer 0/1/2 vs rubric.
3. Disable GrepAI (below), install + index Graphifyy, measure one-time costs.
4. (Arm B, new session) Same queries, sub-agent tools = Bash(graphify) + Read; use ONLY graphify for retrieval. Log arm=B. Score.
5. (Arm C, optional) grep/glob/Read only. Log arm=C.
6. Analyze: per-metric medians by arm; weight ctx_tokens + score (the day-to-day levers); factor in index/maintenance cost and the doc-vs-code split.

Reversible environment changes (per-machine only)

Disable GrepAI (edit .claude/settings.local.json, remove "grepai" from enabledMcpjsonServers; restart session). Re-enable = add it back. Do NOT edit .mcp.json (shared/fleet). Install Graphifyy: py -m pip install graphifyy && graphify install. Uninstall = py -m pip uninstall graphifyy + remove its skill. Snapshot of settings.local.json kept at projects/graphifyy-eval/settings.local.json.bak before any edit.

Open setup unknowns to resolve at install

• Which API key/env var Graphifyy uses for doc/PDF/image ingestion (README didn't say; it bills as "a Claude Code skill"). Confirm before indexing docs so ingest cost is attributable.
• Whether graphify query itself spends LLM tokens to answer (vs returning raw graph context) — affects per-query cost comparison; measure.