search-bots: fix reliability (diagnosed) - gemini 3-retry + grok xsearch auto-fallback to gemini

Mike's must-fix. Diagnosed from RAW output of failing queries (not guessed):
- grok xsearch = TIMEOUT: grok-4.20-multi-agent web_search runs past budget on multi-part queries
  (286s/280s, rc=124, still searching - 183 thoughts, only progress-noise text); buffered json => total loss.
- gemini search = INTERMITTENT empty turn (a clean re-run gave a real 2.6KB answer in 122s); the wrapper
  retried only once, so two empties in a row failed spuriously.

Fixes:
- ask-gemini.sh emit_or_fail: retry up to 3x with 3s/6s backoff (was 1).
- ask-grok.sh xsearch: --output-format streaming-json (salvage partials) + AUTO-FALLBACK to
  ask-gemini.sh search when grok doesn't finish (rc!=0 or empty). Validated e2e: grok timed out
  (rc=124) -> fell back -> gemini returned a real sourced answer (UniFi Teleport invite-link API).

grok's own multi-agent timeout is an xAI-side limitation; the fallback makes xsearch reliable regardless.
Docs: grok SKILL.md xsearch row + CT_THOUGHTS Thought 2 Resolution.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-17 10:38:18 -07:00
parent 58343bd656
commit 315f45bf7c
4 changed files with 63 additions and 39 deletions

View File

@@ -187,31 +187,28 @@ print(r)" < "$OUT"; }
# detect an auth failure in stderr (so we can give a precise remediation hint)
auth_failed() { grep -qiE 'oauth|unauthor|authenticat|login|credential|invalid_grant|401' "$ERR" 2>/dev/null; }
emit_or_fail() { # print .response, or retry once on a transient empty turn, else fail
local txt; txt="$(gresponse)"
if [ -n "$txt" ]; then printf '%s\n' "$txt"; return 0; fi
# Auth failures won't be fixed by a retry — report immediately.
if auth_failed; then
echo "[$SELF] Gemini auth error — run 'gemini' interactively and choose 'Login with Google', then retry." >&2
_logerr "gemini auth/login failure" --context "mode=$MODE"
exit 1
fi
# Gemini occasionally returns an empty turn (or absorbs a 429 backoff into the
# timeout). Replay the identical call once before giving up.
if [ ${#LAST_RUN[@]} -gt 0 ]; then
echo "[$SELF] empty response — retrying once..." >&2
emit_or_fail() { # print .response; gemini intermittently returns an empty turn, so retry a few
# times with backoff before giving up (single retry was insufficient - 2 empties
# in a row caused spurious failures during live research, 2026-06-17).
local txt tries=0 max="${AGY_MAX_TRIES:-3}"
txt="$(gresponse)"
while [ -z "$txt" ]; do
# Auth failures won't be fixed by a retry - report immediately.
if auth_failed; then
echo "[$SELF] Gemini auth error - run 'gemini' interactively and choose 'Login with Google', then retry." >&2
_logerr "gemini auth/login failure" --context "mode=$MODE"; exit 1
fi
tries=$((tries+1))
{ [ "$tries" -ge "$max" ] || [ ${#LAST_RUN[@]} -eq 0 ]; } && break
echo "[$SELF] empty response - retry $tries/$((max-1)) (backoff ${tries}x3s)..." >&2
sleep $((tries*3)) # 3s, 6s, ... backoff (covers transient empties / 429s)
run_gemini "${LAST_RUN[@]}"
txt="$(gresponse)"
if [ -n "$txt" ]; then printf '%s\n' "$txt"; return 0; fi
if auth_failed; then
echo "[$SELF] Gemini auth error — run 'gemini' interactively and choose 'Login with Google', then retry." >&2
_logerr "gemini auth/login failure (after retry)" --context "mode=$MODE"
exit 1
fi
fi
echo "[$SELF] no response from gemini. stderr tail:" >&2
done
if [ -n "$txt" ]; then printf '%s\n' "$txt"; return 0; fi
echo "[$SELF] no response from gemini after $max attempts. stderr tail:" >&2
tail -3 "$ERR" >&2 2>/dev/null || true
_logerr "gemini returned no response (empty after retry)" --context "mode=$MODE err=$(tail -1 "$ERR" 2>/dev/null | tr -d '\n' | cut -c1-80)"
_logerr "gemini returned no response (empty after $max attempts)" --context "mode=$MODE err=$(tail -1 "$ERR" 2>/dev/null | tr -d '\n' | cut -c1-80)"
exit 1
}