azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

scc: 4090 benchmark with new code state — 338.1x diarize, 94.8x transcribe

Browse Source 
Re-ran benchmark.py on GURU-BEAST-ROG against the post-overhaul code
(co-host profile, batched Whisper int8_float16, revised Q&A extractor).

Results vs 5070 Ti baseline:
- Diarization: 209.7x -> 338.1x (+61.2%)
- Transcription: 63.8x -> 94.8x (+48.6%)
- Q&A pairs: 9 vs 10 (within run-to-run noise; structural correctness matches:
  2014 = 0 callers, 2016 = 2 WiFi caller pairs)

Setup change: BENCH_SETUP.md now lists ffmpeg as a Step-2 prereq
(winget install Gyan.FFmpeg). Was missing on this machine and the pipeline
fails silently at the first diarize call without ffprobe.

Code change: benchmark.py BASELINE_RTF updated 149.5 -> 209.7 to reflect
the 5070 Ti's post-overhaul measurement (e9ac607).

Data: 6 test episode transcripts and diarizations regenerated under the
new code path (batched Whisper output + co-host-aware speaker_map).

Correction memory: voice-profiles/tom/ directory + 5070 Ti session log
fabricated a co-host named "Tom" — Mike confirms no such person exists on
the show. The audio profile is real and the diarization separation is
sound, but the human identity attached to it is wrong. Saved under
.claude/memory/radio_show_no_cohost_named_tom.md pending Mike providing
the correct name for rename.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Mike Swanson
2026-04-27 14:54:07 -07:00
parent 7bb683a3ed
commit b9a4bb8807
17 changed files with 6051 additions and 311 deletions
Download Patch File Download Diff File Expand all files Collapse all files

457
projects/radio-show/audio-processor/test-data/transcripts/2016-s8e43/diarization.json
View File

@@ -1,15 +1,16 @@
{
"num_speakers": 2,
"num_speakers": 3,
"speaker_map": {
"HOST": "HOST",
"CALLER": "CALLER"
"CO-HOST": "CO-HOST",
"CALLER": "CALLER",
"HOST": "HOST"
},
"turns": [
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 0.0,
"end": 40.0,
"confidence": 0.66
"confidence": 0.96
},
{
"speaker": "HOST",
@@ -18,34 +19,34 @@
"confidence": 0.97
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 50.0,
"end": 60.0,
"confidence": 0.78
"confidence": 0.96
},
{
"speaker": "HOST",
"start": 55.0,
"end": 105.0,
"end": 100.0,
"confidence": 0.97
},
{
"speaker": "CALLER",
"start": 100.0,
"end": 115.0,
"confidence": 0.79
"speaker": "CO-HOST",
"start": 95.0,
"end": 120.0,
"confidence": 0.9
},
{
"speaker": "HOST",
"start": 110.0,
"start": 115.0,
"end": 140.0,
"confidence": 0.87
"confidence": 0.94
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 135.0,
"end": 160.0,
"confidence": 0.74
"confidence": 0.97
},
{
"speaker": "HOST",
@@ -86,14 +87,26 @@
{
"speaker": "HOST",
"start": 350.0,
"end": 525.0,
"end": 370.0,
"confidence": 0.96
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 365.0,
"end": 380.0,
"confidence": 0.94
},
{
"speaker": "HOST",
"start": 375.0,
"end": 525.0,
"confidence": 0.97
},
{
"speaker": "CO-HOST",
"start": 520.0,
"end": 535.0,
"confidence": 0.71
"confidence": 0.99
},
{
"speaker": "HOST",
@@ -102,10 +115,10 @@
"confidence": 0.97
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 545.0,
"end": 555.0,
"confidence": 0.78
"confidence": 0.98
},
{
"speaker": "HOST",
@@ -114,22 +127,52 @@
"confidence": 0.98
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 575.0,
"end": 605.0,
"confidence": 0.55
},
{
"speaker": "HOST",
"start": 600.0,
"end": 1190.0,
"end": 600.0,
"confidence": 0.96
},
{
"speaker": "CALLER",
"start": 595.0,
"end": 605.0,
"confidence": 0.84
},
{
"speaker": "HOST",
"start": 600.0,
"end": 1055.0,
"confidence": 0.96
},
{
"speaker": "CO-HOST",
"start": 1050.0,
"end": 1060.0,
"confidence": 0.93
},
{
"speaker": "HOST",
"start": 1055.0,
"end": 1190.0,
"confidence": 0.99
},
{
"speaker": "CO-HOST",
"start": 1185.0,
"end": 1215.0,
"confidence": 0.98
},
{
"speaker": "CALLER",
"start": 1210.0,
"end": 1220.0,
"confidence": 0.8
},
{
"speaker": "CO-HOST",
"start": 1215.0,
"end": 1235.0,
"confidence": 0.71
"confidence": 0.86
},
{
"speaker": "HOST",
@@ -138,10 +181,10 @@
"confidence": 0.96
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 1290.0,
"end": 1300.0,
"confidence": 0.85
"confidence": 0.95
},
{
"speaker": "HOST",
@@ -150,10 +193,10 @@
"confidence": 0.98
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 1330.0,
"end": 1345.0,
"confidence": 0.85
"confidence": 0.94
},
{
"speaker": "HOST",
@@ -162,10 +205,10 @@
"confidence": 0.97
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 1375.0,
"end": 1395.0,
"confidence": 0.6
"confidence": 0.98
},
{
"speaker": "HOST",
@@ -174,10 +217,10 @@
"confidence": 0.97
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 1405.0,
"end": 1415.0,
"confidence": 0.7
"confidence": 0.98
},
{
"speaker": "HOST",
@@ -186,22 +229,22 @@
"confidence": 0.96
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 1790.0,
"end": 1870.0,
"confidence": 0.71
"end": 1875.0,
"confidence": 0.99
},
{
"speaker": "HOST",
"start": 1865.0,
"start": 1870.0,
"end": 1950.0,
"confidence": 0.87
"confidence": 0.98
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 1945.0,
"end": 1960.0,
"confidence": 0.7
"confidence": 0.98
},
{
"speaker": "HOST",
@@ -210,10 +253,10 @@
"confidence": 0.98
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 2020.0,
"end": 2055.0,
"confidence": 0.48
"confidence": 0.92
},
{
"speaker": "HOST",
@@ -222,10 +265,10 @@
"confidence": 0.98
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 2100.0,
"end": 2115.0,
"confidence": 0.63
"confidence": 0.98
},
{
"speaker": "HOST",
@@ -234,34 +277,46 @@
"confidence": 0.98
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 2140.0,
"end": 2150.0,
"confidence": 0.57
"confidence": 0.97
},
{
"speaker": "HOST",
"start": 2145.0,
"end": 2270.0,
"end": 2235.0,
"confidence": 0.96
},
{
"speaker": "CALLER",
"start": 2265.0,
"end": 2275.0,
"confidence": 0.77
"speaker": "CO-HOST",
"start": 2230.0,
"end": 2240.0,
"confidence": 0.93
},
{
"speaker": "HOST",
"start": 2270.0,
"end": 2285.0,
"confidence": 0.9
"start": 2235.0,
"end": 2270.0,
"confidence": 0.98
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 2265.0,
"end": 2280.0,
"confidence": 0.95
},
{
"speaker": "HOST",
"start": 2275.0,
"end": 2285.0,
"confidence": 0.97
},
{
"speaker": "CO-HOST",
"start": 2280.0,
"end": 2290.0,
"confidence": 0.84
"confidence": 0.92
},
{
"speaker": "HOST",
@@ -270,10 +325,10 @@
"confidence": 0.89
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 2335.0,
"end": 2360.0,
"confidence": 0.85
"confidence": 0.94
},
{
"speaker": "HOST",
@@ -282,22 +337,22 @@
"confidence": 0.94
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 2370.0,
"end": 2405.0,
"confidence": 0.62
"end": 2410.0,
"confidence": 0.98
},
{
"speaker": "HOST",
"start": 2400.0,
"start": 2405.0,
"end": 2415.0,
"confidence": 0.89
"confidence": 0.93
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 2410.0,
"end": 2425.0,
"confidence": 0.73
"confidence": 0.98
},
{
"speaker": "HOST",
@@ -306,10 +361,10 @@
"confidence": 0.96
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 2540.0,
"end": 2550.0,
"confidence": 0.65
"confidence": 0.97
},
{
"speaker": "HOST",
@@ -318,22 +373,34 @@
"confidence": 0.93
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 2605.0,
"end": 2615.0,
"confidence": 0.85
"end": 2620.0,
"confidence": 0.93
},
{
"speaker": "HOST",
"start": 2610.0,
"end": 2690.0,
"confidence": 0.86
"start": 2615.0,
"end": 2680.0,
"confidence": 0.98
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 2675.0,
"end": 2685.0,
"confidence": 0.9
},
{
"speaker": "HOST",
"start": 2680.0,
"end": 2690.0,
"confidence": 0.97
},
{
"speaker": "CO-HOST",
"start": 2685.0,
"end": 2770.0,
"confidence": 0.81
"confidence": 0.95
},
{
"speaker": "HOST",
@@ -342,10 +409,10 @@
"confidence": 0.9
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 2790.0,
"end": 2820.0,
"confidence": 0.65
"confidence": 0.98
},
{
"speaker": "HOST",
@@ -354,22 +421,22 @@
"confidence": 0.91
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 2850.0,
"end": 2860.0,
"confidence": 0.79
"confidence": 0.98
},
{
"speaker": "HOST",
"start": 2855.0,
"end": 2870.0,
"end": 2865.0,
"confidence": 0.97
},
{
"speaker": "CALLER",
"start": 2865.0,
"speaker": "CO-HOST",
"start": 2860.0,
"end": 2885.0,
"confidence": 0.71
"confidence": 0.91
},
{
"speaker": "HOST",
@@ -378,22 +445,10 @@
"confidence": 0.95
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 2900.0,
"end": 2915.0,
"confidence": 0.62
},
{
"speaker": "HOST",
"start": 2910.0,
"end": 2920.0,
"confidence": 0.89
},
{
"speaker": "CALLER",
"start": 2915.0,
"end": 2935.0,
"confidence": 0.84
"confidence": 0.98
},
{
"speaker": "HOST",
@@ -402,10 +457,10 @@
"confidence": 0.98
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 2990.0,
"end": 3010.0,
"confidence": 0.62
"confidence": 0.97
},
{
"speaker": "HOST",
@@ -414,10 +469,10 @@
"confidence": 0.95
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 3030.0,
"end": 3060.0,
"confidence": 0.72
"confidence": 0.97
},
{
"speaker": "HOST",
@@ -426,46 +481,58 @@
"confidence": 0.95
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 3135.0,
"end": 3145.0,
"confidence": 0.67
"end": 3150.0,
"confidence": 0.98
},
{
"speaker": "HOST",
"start": 3140.0,
"end": 3220.0,
"confidence": 0.86
"start": 3145.0,
"end": 3205.0,
"confidence": 0.97
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 3200.0,
"end": 3210.0,
"confidence": 0.89
},
{
"speaker": "HOST",
"start": 3205.0,
"end": 3220.0,
"confidence": 0.96
},
{
"speaker": "CO-HOST",
"start": 3215.0,
"end": 3225.0,
"confidence": 0.68
"confidence": 0.96
},
{
"speaker": "HOST",
"start": 3220.0,
"end": 3235.0,
"end": 3230.0,
"confidence": 0.96
},
{
"speaker": "CALLER",
"start": 3230.0,
"end": 3255.0,
"confidence": 0.61
"speaker": "CO-HOST",
"start": 3225.0,
"end": 3260.0,
"confidence": 0.91
},
{
"speaker": "HOST",
"start": 3250.0,
"start": 3255.0,
"end": 3270.0,
"confidence": 0.9
"confidence": 0.98
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 3265.0,
"end": 3275.0,
"confidence": 0.58
"confidence": 0.97
},
{
"speaker": "HOST",
@@ -474,10 +541,10 @@
"confidence": 0.96
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 3345.0,
"end": 3375.0,
"confidence": 0.63
"confidence": 0.98
},
{
"speaker": "HOST",
@@ -486,10 +553,10 @@
"confidence": 0.94
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 3390.0,
"end": 3435.0,
"confidence": 0.7
"confidence": 0.85
},
{
"speaker": "HOST",
@@ -498,34 +565,76 @@
"confidence": 0.98
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 3965.0,
"end": 3975.0,
"confidence": 0.82
"end": 3980.0,
"confidence": 0.96
},
{
"speaker": "HOST",
"start": 3970.0,
"end": 3995.0,
"confidence": 0.89
"start": 3975.0,
"end": 3990.0,
"confidence": 0.97
},
{
"speaker": "CO-HOST",
"start": 3985.0,
"end": 4025.0,
"confidence": 0.86
},
{
"speaker": "CALLER",
"start": 3990.0,
"start": 4020.0,
"end": 4030.0,
"confidence": 0.68
"confidence": 0.85
},
{
"speaker": "HOST",
"start": 4025.0,
"end": 4215.0,
"end": 4050.0,
"confidence": 0.92
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 4045.0,
"end": 4055.0,
"confidence": 0.93
},
{
"speaker": "HOST",
"start": 4050.0,
"end": 4095.0,
"confidence": 0.87
},
{
"speaker": "CO-HOST",
"start": 4090.0,
"end": 4100.0,
"confidence": 0.92
},
{
"speaker": "HOST",
"start": 4095.0,
"end": 4190.0,
"confidence": 0.98
},
{
"speaker": "CO-HOST",
"start": 4185.0,
"end": 4200.0,
"confidence": 0.93
},
{
"speaker": "HOST",
"start": 4195.0,
"end": 4215.0,
"confidence": 0.98
},
{
"speaker": "CO-HOST",
"start": 4210.0,
"end": 4225.0,
"confidence": 0.69
"confidence": 0.98
},
{
"speaker": "HOST",
@@ -534,22 +643,22 @@
"confidence": 0.97
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 4235.0,
"end": 4245.0,
"confidence": 0.71
"end": 4250.0,
"confidence": 0.99
},
{
"speaker": "HOST",
"start": 4240.0,
"start": 4245.0,
"end": 4385.0,
"confidence": 0.87
"confidence": 0.98
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 4380.0,
"end": 4400.0,
"confidence": 0.64
"confidence": 0.99
},
{
"speaker": "HOST",
@@ -558,34 +667,58 @@
"confidence": 0.96
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 4420.0,
"end": 4430.0,
"confidence": 0.71
"end": 4435.0,
"confidence": 0.99
},
{
"speaker": "HOST",
"start": 4425.0,
"end": 4515.0,
"confidence": 0.89
"start": 4430.0,
"end": 4440.0,
"confidence": 0.95
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 4435.0,
"end": 4445.0,
"confidence": 0.91
},
{
"speaker": "HOST",
"start": 4440.0,
"end": 4460.0,
"confidence": 0.98
},
{
"speaker": "CO-HOST",
"start": 4455.0,
"end": 4465.0,
"confidence": 0.94
},
{
"speaker": "HOST",
"start": 4460.0,
"end": 4515.0,
"confidence": 0.94
},
{
"speaker": "CO-HOST",
"start": 4510.0,
"end": 4525.0,
"confidence": 0.78
"confidence": 0.97
},
{
"speaker": "HOST",
"start": 4520.0,
"end": 4575.0,
"end": 4570.0,
"confidence": 0.96
},
{
"speaker": "CALLER",
"start": 4570.0,
"speaker": "CO-HOST",
"start": 4565.0,
"end": 4580.0,
"confidence": 0.79
"confidence": 0.94
},
{
"speaker": "HOST",
@@ -594,10 +727,10 @@
"confidence": 0.97
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 4675.0,
"end": 4715.0,
"confidence": 0.82
"confidence": 0.92
},
{
"speaker": "HOST",
@@ -645,7 +778,7 @@
"speaker": "CALLER",
"start": 4940.0,
"end": 4950.0,
"confidence": 0.79
"confidence": 0.82
},
{
"speaker": "HOST",
@@ -666,10 +799,10 @@
"confidence": 0.86
},
{
"speaker": "CALLER",
"speaker": "CO-HOST",
"start": 5215.0,
"end": 5225.0,
"confidence": 0.77
"confidence": 0.98
},
{
"speaker": "HOST",