claudetools/projects/radio-show/audio-processor/session-logs/2026-04-27-4090-benchmark-and-test-set.md

Session Log — 2026-04-27 (continuation)

Project: The Computer Guru Show — Archive Mining System Goal: RTX 4090 perf comparison + run unseen test episodes through full pipeline (transcribe / diarize / Q&A) Machine: GURU-BEAST-ROG (RTX 4090, 24GB) User: Mike Swanson (mike)

Companion to:

• 2026-04-27-diarization-pipeline.md (DESKTOP-0O8A1RL, RTX 5070 Ti — initial diarization fixes)
• 2026-04-27-qa-extraction-cohost-indexing.md (DESKTOP-0O8A1RL — co-host profile, batched Whisper, Q&A overhaul)

This run uses the post-overhaul code (commit e9ac607): batched Whisper transcription, co-host-aware diarizer, revised Q&A extractor.

---

Headline

Metric	5070 Ti baseline	RTX 4090	Delta
Diarization	209.7x realtime	338.1x	+128.4x (+61.2%)
Transcription (batched, large-v3 int8_float16)	63.8x	94.8x	+31.0x (+48.6%)
Q&A pairs (6 test episodes)	10	9	within noise

21,374s of audio (5h 56m) end-to-end on the 4090: 225.5s transcription + 63.2s diarization + Q&A extraction.

---

Important — "Tom" co-host name is wrong

The 5070 Ti session built a voice profile labeled voice-profiles/tom/ and described it in the session log as "Tom, regular in-studio co-host/board-op roughly 2013-2016." Mike confirmed on this session: there is no co-host named Tom. The voice profile is real (clean cosine separation, 0.698 vs Mike) and the diarization correctly identifies the second speaker, but the human identity attached to it is hallucinated.

The directory, profiles.json entry, build_cohost_profile.py references, and the 5070 Ti session log all carry the bogus name. Identity TBD pending Mike confirming who that voice actually is.

Memory entry added: .claude/memory/radio_show_no_cohost_named_tom.md. The profile will be renamed once Mike provides the correct identity.

---

Setup notes (for next machine)

• ffmpeg/ffprobe is required on PATH — the voice profiler shells out to ffprobe for audio duration and the pipeline crashes on the first diarize call without it. Was missing on this machine; installed via winget install Gyan.FFmpeg. BENCH_SETUP.md updated to call this out as a Step-2 prereq.
• .gitignore (added in e9ac607) excludes episodes/, transcripts/, *.db, .venv. The test MP3s + transcripts I committed earlier in 2c06e72 are still tracked from before the gitignore arrived; can be git rm --cached-ed in a follow-up cleanup.
• All voice profiles, training data, and test MP3s were already on this machine via prior auto-sync.

---

Phase 1 — Whisper Transcription (large-v3, batched, int8_float16, batch_size=16)

Episode	Audio	Wall	RTF
2011-03-12-hr1	2509s	29.7s	84.6x
2012-03-10-hr1	2634s	30.3s	87.0x
2012-06-09-hr1	2648s	33.6s	78.8x
2014-s6e19	2914s	30.2s	96.6x
2016-s8e43	5326s	49.2s	108.2x
2017-s9e30	5343s	52.5s	101.8x
Total	21374s	225.5s	94.8x

vs 5070 Ti's 63.8x: +48.6%.

Batching is doing real work here. The pre-batched code path on this same hardware (first benchmark run earlier today) was 14.8x — batching gave a 6.4× speedup on the 4090.

---

Phase 2 — Diarization (with co-host profile applied)

Episode	Audio	Wall	RTF	Turns	HOST	CALLER
2011-03-12-hr1	2509s	9.1s	275.0x	25	2455s	70s
2012-03-10-hr1	2634s	7.6s	348.3x	22	2615s	90s
2012-06-09-hr1	2648s	7.7s	343.1x	13	2500s	10s
2014-s6e19	2914s	8.3s	352.6x	31	2625s	30s
2016-s8e43	5326s	15.1s	353.6x	134	4615s	140s
2017-s9e30	5343s	15.5s	345.1x	69	4945s	350s
Total	21374s	63.2s	338.1x	294	19755s	690s

vs 5070 Ti baseline: 209.7x → 338.1x (+61.2%).

Per-episode RTFs cluster tightly at 343-354x for warm episodes (5/6); episode 1 carries the cold-start penalty at 275.0x. Apples-to-apples vs the 5070 Ti measurement which also includes a cold start.

Aggregate CALLER time dropped from 2665s (pre-co-host pipeline, run earlier today) to 690s. That ~2000s delta is the second-voice signal correctly being routed away from the CALLER bucket. The benchmark table only sums HOST + CALLER, so CO-HOST seconds aren't shown in the totals — present in the per-episode diarization.json files.

---

Phase 3 — Q&A Extraction (post-overhaul: turn-based lookback, 4s CALLER preference, expanded promo signatures)

Episode	4090 Q&A pairs	5070 Ti reference	Note
2011-03-12-hr1	1	3	-2
2012-03-10-hr1	2	1	+1
2012-06-09-hr1	0	1	-1
2014-s6e19	0	0	match (gaming, no callers)
2016-s8e43	2	2	match (WiFi caller)
2017-s9e30	4	3	+1
Total	9	10	-1

Differences are within noise. Likely sources:

• Whisper batched inference produces slightly different segment boundaries on identical audio under different GPU schedule orderings.
• Sliding-window diarization midpoint resolution can put a borderline segment in either bucket on different runs.
• Q&A extraction thresholds are sensitive to small boundary shifts.

The two structural correctness signals match: 2014 = 0 (no callers in gaming special) and 2016 = 2 (real WiFi caller, two-turn). That's the meaningful test. Aggregate ±1 across six episodes is acceptable run-to-run drift.

---

Files written / modified

• test-data/transcripts/<stem>/transcript.json (6, regenerated with batched Whisper)
• test-data/transcripts/<stem>/diarization.json (6, regenerated with co-host-aware diarizer)
• benchmark.py line 27 — BASELINE_RTF updated 149.5 → 209.7
• BENCH_SETUP.md — added ffmpeg prereq to Step 2
• .claude/memory/radio_show_no_cohost_named_tom.md (new, project memory)
• .claude/memory/MEMORY.md (index updated)

archive.db is not on this machine — index update happens on DESKTOP-0O8A1RL.

---

Pending work (from 5070 Ti session, still unblocked)

1. Resolve "Tom" identity — Mike to confirm who the second voice is in 2014-s6e19 and 2016-s8e43. Then rename voice-profiles/tom/, update profiles.json, fix labels in code. Until then, voice-profile data is correct but mislabeled.
2. Full archive download — 579 MP3s from IX server (~30-40GB). 4090 + Tailscale ready.
3. Full pipeline run on archive — at 338x diarization + 95x transcription, total wall time for ~30h of audio extrapolates to roughly 19 minutes diarization + 19 minutes transcription. Disk I/O may dominate.

---

Note for Mike

• "Tom" is wrong — see callout above. Tell me who that is and I'll do the rename in one pass (directory, profiles.json, build_cohost_profile.py, the 5070 Ti session log, and a fresh diarization pass to update speaker_map).
• BENCH_SETUP.md got a one-paragraph ffmpeg prereq added at the top of Step 2.