claudetools/projects/radio-show/audio-processor/session-logs/2026-04-27-4090-benchmark-and-test-set.md

Session Log — 2026-04-27 (continuation)

Project: The Computer Guru Show — Archive Mining System Goal: RTX 4090 perf comparison + run unseen test episodes through full pipeline (transcribe / diarize / Q&A) Machine: GURU-BEAST-ROG (RTX 4090, 24GB) User: Mike Swanson (mike)

Companion to:

• 2026-04-27-diarization-pipeline.md (DESKTOP-0O8A1RL, RTX 5070 Ti — initial diarization fixes)
• 2026-04-27-qa-extraction-cohost-indexing.md (DESKTOP-0O8A1RL — co-host profile, batched Whisper, Q&A overhaul)

This run uses the post-overhaul code (commit e9ac607): batched Whisper transcription, co-host-aware diarizer, revised Q&A extractor.

---

Headline

Metric	5070 Ti baseline	RTX 4090	Delta
Diarization	209.7x realtime	338.1x	+128.4x (+61.2%)
Transcription (batched, large-v3 int8_float16)	63.8x	94.8x	+31.0x (+48.6%)
Q&A pairs (6 test episodes)	10	9	within noise

21,374s of audio (5h 56m) end-to-end on the 4090: 225.5s transcription + 63.2s diarization + Q&A extraction.

---

Co-host identity correction — Tara, not Tom

The 5070 Ti session fabricated a co-host named "Tom" — Mike confirmed there is no such person on the show. After listening to the source windows, Mike identified the voice in both 2014-s6e19 and 2016-s8e43 as Tara (a real co-host; the show has had multiple over the years).

Rename swept this session:

• voice-profiles/tom/ → voice-profiles/tara/ (git mv, all 44 embeddings + composite preserved)
• voice-profiles/profiles.json: "Tom" key → "Tara"
• build_cohost_profile.py: docstring, TOM_WINDOWS → TARA_WINDOWS, COHOST_NAME = "Tara", console output strings
• projects/radio-show/session-logs/2026-04-27-qa-extraction-cohost-indexing.md: correction header added, all body references updated
• .claude/memory/radio_show_no_cohost_named_tom.md: resolution recorded
• Diarization re-run post-rename so speaker_map in each diarization.json emits Cohost: Tara

The 5070 Ti session log's claim of "Tom was the regular co-host roughly 2013-2016" carried two errors: the wrong name AND an unverified tenure window. The corrected log notes Tara appears in 2014-s6e19 and 2016-s8e43 only — generalizing to the full 2013-2016 era hasn't been confirmed.

---

Setup notes (for next machine)

• ffmpeg/ffprobe is required on PATH — the voice profiler shells out to ffprobe for audio duration and the pipeline crashes on the first diarize call without it. Was missing on this machine; installed via winget install Gyan.FFmpeg. BENCH_SETUP.md updated to call this out as a Step-2 prereq.
• .gitignore (added in e9ac607) excludes episodes/, transcripts/, *.db, .venv. The test MP3s + transcripts I committed earlier in 2c06e72 are still tracked from before the gitignore arrived; can be git rm --cached-ed in a follow-up cleanup.
• All voice profiles, training data, and test MP3s were already on this machine via prior auto-sync.

---

Phase 1 — Whisper Transcription (large-v3, batched, int8_float16, batch_size=16)

Episode	Audio	Wall	RTF
2011-03-12-hr1	2509s	29.7s	84.6x
2012-03-10-hr1	2634s	30.3s	87.0x
2012-06-09-hr1	2648s	33.6s	78.8x
2014-s6e19	2914s	30.2s	96.6x
2016-s8e43	5326s	49.2s	108.2x
2017-s9e30	5343s	52.5s	101.8x
Total	21374s	225.5s	94.8x

vs 5070 Ti's 63.8x: +48.6%.

Batching is doing real work here. The pre-batched code path on this same hardware (first benchmark run earlier today) was 14.8x — batching gave a 6.4× speedup on the 4090.

---

Phase 2 — Diarization (with co-host profile applied)

Episode	Audio	Wall	RTF	Turns	HOST	CALLER
2011-03-12-hr1	2509s	9.1s	275.0x	25	2455s	70s
2012-03-10-hr1	2634s	7.6s	348.3x	22	2615s	90s
2012-06-09-hr1	2648s	7.7s	343.1x	13	2500s	10s
2014-s6e19	2914s	8.3s	352.6x	31	2625s	30s
2016-s8e43	5326s	15.1s	353.6x	134	4615s	140s
2017-s9e30	5343s	15.5s	345.1x	69	4945s	350s
Total	21374s	63.2s	338.1x	294	19755s	690s

vs 5070 Ti baseline: 209.7x → 338.1x (+61.2%).

Per-episode RTFs cluster tightly at 343-354x for warm episodes (5/6); episode 1 carries the cold-start penalty at 275.0x. Apples-to-apples vs the 5070 Ti measurement which also includes a cold start.

Aggregate CALLER time dropped from 2665s (pre-co-host pipeline, run earlier today) to 690s. That ~2000s delta is the second-voice signal correctly being routed away from the CALLER bucket. The benchmark table only sums HOST + CALLER, so CO-HOST seconds aren't shown in the totals — present in the per-episode diarization.json files.

---

Phase 3 — Q&A Extraction (post-overhaul: turn-based lookback, 4s CALLER preference, expanded promo signatures)

Episode	4090 Q&A pairs	5070 Ti reference	Note
2011-03-12-hr1	1	3	-2
2012-03-10-hr1	2	1	+1
2012-06-09-hr1	0	1	-1
2014-s6e19	0	0	match (gaming, no callers)
2016-s8e43	2	2	match (WiFi caller)
2017-s9e30	4	3	+1
Total	9	10	-1

Differences are within noise. Likely sources:

• Whisper batched inference produces slightly different segment boundaries on identical audio under different GPU schedule orderings.
• Sliding-window diarization midpoint resolution can put a borderline segment in either bucket on different runs.
• Q&A extraction thresholds are sensitive to small boundary shifts.

The two structural correctness signals match: 2014 = 0 (no callers in gaming special) and 2016 = 2 (real WiFi caller, two-turn). That's the meaningful test. Aggregate ±1 across six episodes is acceptable run-to-run drift.

---

Files written / modified

• test-data/transcripts/<stem>/transcript.json (6, regenerated with batched Whisper)
• test-data/transcripts/<stem>/diarization.json (6, regenerated with co-host-aware diarizer)
• benchmark.py line 27 — BASELINE_RTF updated 149.5 → 209.7
• BENCH_SETUP.md — added ffmpeg prereq to Step 2
• .claude/memory/radio_show_no_cohost_named_tom.md (new, project memory)
• .claude/memory/MEMORY.md (index updated)

archive.db is not on this machine — index update happens on DESKTOP-0O8A1RL.

---

Tara distribution across the test set (post-rename diarization)

After the rename, the diarizer's per-episode speaker_map shows Tara in all 6 test episodes — well beyond the 2014+2016 the 5070 Ti session log claimed.

Episode	Tara (seconds)	% of audio	Read
2011-03-12-hr1	140s (2:20)	5.6%	likely false positive — Mike confirms 2011 was pure call-in
2012-03-10-hr1	30s (0:30)	1.1%	likely false positive — 2012 was pure call-in
2012-06-09-hr1	340s (5:40)	12.8%	suspicious — too much for noise; awaiting Mike confirm
2014-s6e19	680s (11:20)	23.3%	confirmed (Mike)
2016-s8e43	1890s (31:30)	35.5%	confirmed (Mike)
2017-s9e30	610s (10:10)	11.4%	plausible — pending Mike confirm; 5070 Ti log only listed Tara in 2014+2016

Mike's broader correction (2026-04-27):

• Co-hosts rotated through over the years. Confirmed: Tara, Randall (early years), Rob (early years + occasional 2018/2019).
• Producers / board ops would sometimes go on-air. Named so far: Andrew, Shannon, Ken, plus "a couple more" Mike doesn't recall off-hand.

Of all these, only Tara has a voice profile. Every other co-host AND every producer-on-air moment in the archive is currently being labeled CALLER, which inflates Q&A false positives in those eras and episodes.

The small Tara percentages in 2011/2012 (1-13%) most likely reflect the 0.85 cosine threshold hitting on a similar-sounding speaker that isn't actually Tara — could be a producer (Andrew/Shannon/Ken/etc) or another early-years voice we haven't catalogued. Worth Mike sampling these short windows to identify before assuming false positive vs producer.

Implication for full-archive runs: before processing the 579-episode archive in earnest, build profiles for at least Randall, Rob, and the named producers. Otherwise the Q&A extraction across early-years and 2018/2019 episodes will inherit the same false-positive pattern that originally produced 12 bogus pairs in 2016-s8e43.

---

Pending work (from 5070 Ti session, still unblocked)

1. Resolve "Tom" identity — Mike to confirm who the second voice is in 2014-s6e19 and 2016-s8e43. Then rename voice-profiles/tom/, update profiles.json, fix labels in code. Until then, voice-profile data is correct but mislabeled.
2. Full archive download — 579 MP3s from IX server (~30-40GB). 4090 + Tailscale ready.
3. Full pipeline run on archive — at 338x diarization + 95x transcription, total wall time for ~30h of audio extrapolates to roughly 19 minutes diarization + 19 minutes transcription. Disk I/O may dominate.

---

Note for Mike

• "Tom" is wrong — see callout above. Tell me who that is and I'll do the rename in one pass (directory, profiles.json, build_cohost_profile.py, the 5070 Ti session log, and a fresh diarization pass to update speaker_map).
• BENCH_SETUP.md got a one-paragraph ffmpeg prereq added at the top of Step 2.