ct-thoughts: web-search bots reliability = MUST FIX (Mike) + research-method correction

Mike's correction: web search (grok xsearch + gemini search) carries at least as much weight as
live API probing - the searches gave the real leads this session (connector proxy, teleport setting
path); blind endpoint-probing is "highly suspect" (mostly 404s). And the search bots MUST be properly
fixed - both returned empty repeatedly on UniFi research despite the same-day partial grok fix.

- docs/CT_THOUGHTS.md: Thought 2 (HIGH PRIORITY) - web-search reliability must-fix, with the observed
  failures + a proper-fix investigation plan (capture failing-query JSON; max-turns/streaming-json/
  retry; cross-fallback grok<->gemini; 5/5 acceptance).
- memory feedback_web_search_over_probing: lead with web search/docs; probe only to CONFIRM a
  hypothesis, never as primary discovery. Reading our own config is fine; guessing paths is not.
- errorlog correction logged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-17 09:36:35 -07:00
parent 8f72178d8a
commit 1dd2f208a0
4 changed files with 92 additions and 0 deletions

View File

@@ -57,6 +57,7 @@
- [/tmp path mismatch on Windows](feedback_tmp_path_windows.md) — Write tool and Git Bash resolve `/tmp` to DIFFERENT real dirs. Use heredoc or workspace path for JSON payloads handed to curl.
- [Windows strips embedded double-quotes](feedback_windows_quote_stripping.md) — Embedded `"` in an arg gets eaten twice over: PowerShell->curl.exe (CommandLineToArgvW) AND RMM->cmd.exe. Use single-quoted heredoc `<<'JSON'` + `--data-binary @-` for bodies; build `"` from `[char]34`; or drop the quoted part (e.g. `shutdown /c`).
- [Interview the AI / read its docs before probing](feedback_interview_ai_read_docs.md) — To learn an external AI/CLI's syntax or capabilities, READ its bundled docs (Grok: `~/.grok/docs/user-guide/`, `README.md`, `grok inspect`/`models`/`--help`) or interview the model; don't guess flags or run slow trial-and-error. One run to confirm a doc-derived hypothesis, not a dozen to discover.
- [Web search over blind probing](feedback_web_search_over_probing.md) — For external API/capability discovery, LEAD with web search (grok/gemini) + vendor docs; live endpoint-probing only CONFIRMS a hypothesis, never the primary discovery method (it mostly 404s, "highly suspect"). Reading a system's OWN config is fine; guessing unknown PATHS is not. Web-search bots being flaky is a must-fix (CT_THOUGHTS Thought 2).
- [Windows bash command mapping](feedback_windows_bash_mapping.md) — `bash` often resolves to WSL stub instead of Git/MSYS bash required by the harness. Fix by prepending `C:\Program Files\Git\bin` (and usr\bin) to PATH, or source `.claude/scripts/ensure-git-bash.ps1`. Profile has the logic; use plain `bash .claude/scripts/...` after remap. See the helper and this memory file for details.
- [Git must authenticate non-interactively](feedback_git_noninteractive_auth.md) — Mike's gripe with Git for Windows is the constant password prompts (GCM) that hang automation, NOT the tool itself. D:\ClaudeTools is set to `credential.helper=store` primed with the azcomputerguru Gitea API token (host 172.16.3.20:3000); always set `GIT_TERMINAL_PROMPT=0`. Any never-prompts solution is acceptable.
- [Vault git auth — GCM shadows store token](feedback_vault_gcm_shadow_auth.md) — vault sync "Failed to authenticate user" on git.azcomputerguru.com: GCM is first in the helper chain and shadows the valid store token. Fix (machine-local): store-only credential.helper reset + pin `azcomputerguru@` in the vault remote URL so store returns the durable PAT (not the volatile OAUTH_USER JWT). Applied GURU-5070 2026-06-07.

View File

@@ -0,0 +1,29 @@
---
name: feedback_web_search_over_probing
description: For external API/capability discovery, LEAD with web search (grok/gemini) and bundled docs; use live endpoint-probing only to CONFIRM a search/doc-derived hypothesis - never as the primary discovery method. Mike's correction 2026-06-17.
metadata:
type: feedback
---
When figuring out an external system's API surface or capabilities, **web search (grok
xsearch / gemini search) and the vendor's own docs carry AT LEAST as much weight as live
experimentation** - usually more.
**Why (Mike, 2026-06-17):** blind endpoint-probing ("does `/stat/openvpn` exist? does
`/cmd/vpnmgr`?") is guessing - it mostly 404s and is "highly suspect" as a source of truth.
The genuinely valuable leads this session came from the searches: grok surfaced the UniFi
**cloud connector proxy** (`/v1/connector/consoles/.../proxy/...`); gemini surfaced the
**Teleport `/rest/setting/teleport`** path. Probing only *confirmed* those after the search
pointed the way.
**How to apply:**
- Discovery order: web search + bundled docs FIRST -> form a specific hypothesis -> then ONE
targeted live call to CONFIRM it. Not: spray candidate URLs and infer from status codes.
- Reading a system's OWN config (e.g. our gateway's `networkconf`) is fine - that's reading
real data, not guessing endpoints. The "suspect" part is guessing unknown PATHS.
- Do not present probe results as "authoritative" over web-search findings; weight them at
least equally and reconcile.
- Corollary: the web-search bots being flaky is a real liability (see CT_THOUGHTS "Thought 2 -
web-search reliability MUST FIX"); when they fail, say so plainly rather than silently
falling back to guessing and calling it authoritative.
- Complements [[feedback_interview_ai_read_docs]].