# Session Log: 2026-03-21

## Session Summary

Continued work on fixing the terrible speaker sound quality on the Lenovo Legion Pro 7 16IAX10H (CachyOS workstation). Previous session (2026-03-20) diagnosed the root cause and applied initial fixes (forced SOF driver, EasyEffects EQ). This session discovered that the SOF driver alone was insufficient because the **Awinic AW88399 smart amplifiers** were not integrated into the audio pipeline, and built a custom kernel with the community patch to fix it.

### Key Decisions
- Used the community patch from `nadimkobeissi/16iax10h-linux-sound-saga` GitHub repo rather than attempting to write our own driver
- Built a full CachyOS kernel package (not DKMS) because the patch modifies existing kernel source files (ACPI scan, serial-multi-instantiate, alc269.c, aw88399.c ASoC driver)
- Installed UCM2 config modifications for proper speaker routing
- Used BUILDDIR on main disk after tmpfs (16GB) ran out of space during kernel extraction

### Problems Encountered
1. **tmpfs too small** - Initial `makepkg` in `/tmp` failed with "No space left on device" because the kernel source + build artifacts exceed 16GB. Fixed by moving build to `/home/guru/kernel-build/` with `BUILDDIR` override.
2. **Missing build deps** - `bc`, `cpio`, `rust`, `rust-bindgen`, `rust-src` needed installation.

## Hardware Details

- **Laptop**: Lenovo Legion Pro 7 16IAX10H (DMI product: 83F5)
- **Audio Controller**: Intel 800 Series ACE (PCI `80:1f.3`), PCI SSID `17aa:3d6c`
- **Codec**: Realtek ALC287, codec SSID `17aa:3906`
- **Smart Amplifier**: Awinic AW88399 on I2C (ACPI: `AWDZ8399:00`, path: `\_SB_.PC02.I2C2.SPKR`)
- **Kernel**: 6.19.9-1-cachyos with SOF driver (`sof-audio-pci-intel-mtl`)
- **SOF Firmware**: `intel/sof-ipc4/arl-s/sof-arl-s.ri` v2.14.1.1
- **SOF Topology**: `intel/sof-ipc4-tplg/sof-hda-generic.tplg`

## Root Cause Analysis

The sound was terrible because:
1. **SOF driver loaded but using generic HDA topology** - no smart amp integration
2. **AW88399 kernel module loaded with 0 references** - `snd_soc_aw88399` was loaded but nothing used it because there was no HDA side-codec bridge driver
3. **Wrong codec fixup** - Kernel picked fixup for PCI SSID `17aa:0000` (generic/none) instead of `17aa:3906` because the PCI subsystem ID is actually `17aa:3d6c` which had no quirk entry
4. **No HDA side-codec bridge** - The stock kernel has `CONFIG_SND_SOC_AW88399=m` (ASoC driver) but NOT `CONFIG_SND_HDA_SCODEC_AW88399` (the HDA bridge driver that connects ALC287 to the I2C amps) - this driver doesn't exist in mainline Linux as of kernel 6.19

## Fix Applied

### Source: Community Patch
- Repository: https://github.com/nadimkobeissi/16iax10h-linux-sound-saga
- Patch: `16iax10h-audio-linux-6.19.8.patch` (979 lines, 10 files)
- Related issues: CachyOS #687, #707; Kernel Bugzilla #218329
- Status: NOT upstreamed to mainline kernel; $2,000 bounty exists for upstream submission

### What the Patch Does
1. **Creates new HDA side-codec bridge driver** (`snd-hda-scodec-aw88399`, `snd-hda-scodec-aw88399-i2c`) - connects ALC287 HDA codec to AW88399 amps via I2C
2. **Adds ACPI ignore entry** for `AWDZ8399` in `drivers/acpi/scan.c` so serial-multi-instantiate handles it
3. **Adds serial-multi-instantiate node** for `aw88399-hda` with 2 instances
4. **Adds HDA fixup entries** in `alc269.c`: `ALC287_FIXUP_LENOVO_LEGION_AW88399` for SSIDs `17aa:3906`, `17aa:3907`, `17aa:3938`
5. **Exports functions** from ASoC `aw88399.c`: `aw88399_start()`, `aw88399_stop()`, `aw88399_init()`, `aw88399_request_firmware_file()`
6. **Removes ACPI match** from ASoC driver to prevent binding conflict with HDA side-codec
7. **DMI-based L/R channel swap** for products 83F5, 83RU, 83F4 (hardware wiring issue)

### Files Created/Modified on System

#### Firmware
- **Created**: `/lib/firmware/aw88399_acf.bin` (amplifier calibration firmware from Windows driver)

#### UCM2 Configs
- **Backed up**: `/usr/share/alsa/ucm2/HDA/HiFi-analog.conf.bak`
- **Backed up**: `/usr/share/alsa/ucm2/HDA/HiFi-mic.conf.bak`
- **Modified**: `/usr/share/alsa/ucm2/HDA/HiFi-analog.conf` (from community repo)
- **Modified**: `/usr/share/alsa/ucm2/HDA/HiFi-mic.conf` (from community repo)

#### Kernel Build
- **Build directory**: `/home/guru/kernel-build/`
- **PKGBUILD**: Modified CachyOS PKGBUILD with `aw88399-hda-scodec.patch` added to source array and `CONFIG_SND_HDA_SCODEC_AW88399=m`, `CONFIG_SND_HDA_SCODEC_AW88399_I2C=m` enabled
- **Built packages**:
  - `/home/guru/kernel-build/linux-cachyos-6.19.9-1-x86_64.pkg.tar.zst` (147MB)
  - `/home/guru/kernel-build/linux-cachyos-headers-6.19.9-1-x86_64.pkg.tar.zst` (37MB)

#### Kernel Installed
- Packages installed via `sudo pacman -U`
- Initramfs regenerated automatically
- Limine bootloader config updated automatically

### Previous Session Fixes Still Active
- `/etc/modprobe.d/sof-force.conf` - forces SOF driver (`dsp_driver=3`)
- `~/.local/share/easyeffects/output/Legion-Speakers-v2.json` - EasyEffects EQ preset
- EasyEffects sink set as default audio output

## Configuration Changes

### Packages Installed This Session
- `bc` (kernel build dep)
- `cpio` (kernel build dep)
- `rust` (kernel build dep - LTO kernel)
- `rust-bindgen` (kernel build dep)
- `rust-src` (kernel build dep)

### Kernel Config Added
```
CONFIG_SND_HDA_SCODEC_AW88399=m
CONFIG_SND_HDA_SCODEC_AW88399_I2C=m
```

## Update: 05:30 — Post-Reboot Tuning & Volume Hotkey Fix

### Session Summary
Booted into the patched kernel from the previous session. AW88399 amps bound successfully. Addressed two issues: (1) volume hotkeys not working, (2) sound quality issues (over-driven, then lacking clarity). Also wrote a comprehensive forum post documenting the full fix.

### Post-Reboot Verification Results
All successful:
- AW88399 HDA side-codec: Both amps registered and bound (channels 0 and 1)
- Modules loaded: `snd_hda_scodec_aw88399`, `snd_hda_scodec_aw88399_i2c`, `snd_soc_aw88399` all with references > 0
- Correct fixup: `ALC287: picked fixup for codec SSID 17aa:3906`
- Autoconfig: `line_outs=2 (0x14/0x17) type:speaker` — both speaker pins active
- I2C amps respond at addresses `0x34` and `0x35` (chip ID `0x8321` = AW88399)
- Firmware: `aw88399_acf.bin` loads silently (no dmesg output on success)

### Key Decisions Made This Session

1. **Removed EasyEffects from audio pipeline** — It was creating a virtual PipeWire sink which broke volume hotkeys AND was applying heavy DSP (bass enhancer, exciter, 15-band EQ, loudness, multiband compressor, limiter) that over-drove the sound now that the amps are working properly.

2. **Removed UCM2 Post Mixer Analog override** — The community UCM2 config had a Legion-specific block that routed PipeWire volume control through the SOF DSP's `Post Mixer Analog Playback Volume` (numid=40). This control has coarse 2dB steps and a -90dB to 0dB range. At typical 50% PipeWire volume, this applied -18dB of digital attenuation in the DSP, destroying dynamic range and clarity. Removing it lets PipeWire use the HDA codec's `Speaker Playback Volume` which has fine 0.75dB steps.

3. **Set all SOF DSP volumes to 0dB** — Pre Mixer, Post Mixer, and Deepbuffer all maxed so no intermediate digital attenuation.

### Problems Encountered & Solutions

#### Problem 1: Volume Hotkeys Not Working
**Symptom:** Fn+Volume Up/Down keys did nothing. Tray slider worked fine.

**Diagnosis journey:**
- Verified keys generate `KEY_VOLUMEUP`/`KEY_VOLUMEDOWN` via evtest on the ITE Device(8258) keyboard (event4)
- KDE's `kmix` shortcuts were correctly registered (`increase_volume=Volume Up`, etc.)
- `invokeShortcut "mute"` worked via DBus, but `invokeShortcut "increase_volume"` did nothing
- Discovered the issue is **NOT the keyboard** — it's plasma-pa's `audioshortcutsservice` KDED module

**Root cause:** plasma-pa's `PreferredDevice::sink()` doesn't handle virtual/combined sinks. When EasyEffects virtual sink is the default, `increase_volume` and `decrease_volume` handlers silently fail. Mute works because it uses a different code path. Related: [KDE Bug 456310](https://bugs.kde.org/show_bug.cgi?id=456310).

**Fix:** Set hardware speaker sink as default instead of EasyEffects virtual sink:
```bash
pactl set-default-sink alsa_output.pci-0000_80_1f.3-platform-skl_hda_dsp_generic.HiFi__Speaker__sink
pkill easyeffects
```

**Note:** This affects ALL virtual sinks, including PipeWire filter-chain sinks. Any `media.class = Audio/Sink` filter will have the same hotkey problem on KDE Plasma 6 Wayland.

#### Problem 2: Over-Driven Sound
**Symptom:** Sound was a bit distorted/over-driven after removing EasyEffects.

**Cause:** ALSA Master and Speaker volumes were at 100% (0dB) while PipeWire was also at 100%. Combined with the AW88399 amp gain, the signal was clipping.

**Fix:** Let PipeWire control volume through the Speaker DAC, keep ALSA levels at 0dB reference.

#### Problem 3: Lacking Clarity / "Paper Over Speakers" Sound
**Symptom:** After fixing over-drive, sound lacked center/clarity/spatial definition.

**Root cause:** The community UCM2 config's Legion override was routing PipeWire volume through `Post Mixer Analog Playback Volume` (SOF DSP). At 50% PipeWire volume → Post Mixer at 36/45 → -18dB digital attenuation with coarse 2dB steps. This was killing dynamic range before the signal even reached the DAC/amps.

**Fix:** Removed the Legion override block (lines 130-147) from `/usr/share/alsa/ucm2/HDA/HiFi-analog.conf`:
```
If.legion {
    Condition { Type String; Haystack ...; Needle "83F5" }
    True.If.postmixer {
        ...Post Mixer Analog override...
    }
}
```

Set SOF DSP volumes to max:
```bash
amixer -c 1 cset numid=39 45,45  # Pre Mixer
amixer -c 1 cset numid=40 45,45  # Post Mixer
amixer -c 1 cset numid=51 45,45  # Deepbuffer
systemctl --user restart pipewire pipewire-pulse wireplumber
```

#### Problem 4: Speaker Balance — Tweeters vs Woofers
**Discovery:** The laptop has two speaker pairs:
- **Pin 0x14** ("Speaker") → Tweeters near screen hinge, driven directly by ALC287 DAC 0x02
- **Pin 0x17** ("Bass Speaker") → Woofers near palmrest, driven through AW88399 smart amps via DAC 0x06

Initially only tweeters played (terrible thin sound). After the kernel patch, the woofers now play through the amps. The tweeters are quieter than the woofers because the amps add significant gain. Both play simultaneously — the woofers just dominate.

**Current state:** Balanced and sounds good per user feedback. Both speaker pairs confirmed working (tested by muting Bass Speaker to isolate tweeters).

### Configuration Changes This Session

#### Files Modified
- `/usr/share/alsa/ucm2/HDA/HiFi-analog.conf` — Removed Legion Post Mixer Analog override (lines 130-147)
- `~/.config/autostart/com.github.wwmm.easyeffects.desktop` — Added `Hidden=true` to disable autostart
- `/etc/udev/hwdb.d/90-ite-keyboard-fix.hwdb` — Created (hwdb rule for ITE keyboard, may not be needed now)

#### Packages Installed
- `evtest` — For diagnosing keyboard input events
- `wtype` — For Wayland key simulation testing (not ultimately needed)

#### Services Changed
- EasyEffects: Stopped and autostart disabled
- PipeWire/WirePlumber: Restarted multiple times during tuning

### Final Working Audio Configuration
```
Kernel: 6.19.9-1-cachyos (patched with AW88399 HDA side-codec)
Default sink: alsa_output.pci-0000_80_1f.3-platform-skl_hda_dsp_generic.HiFi__Speaker__sink
Volume control: PipeWire → Speaker Playback Volume (DAC 0x02, 0.75dB steps)
ALSA Master: 100% (0dB)
ALSA Speaker: Controlled by PipeWire
ALSA Bass Speaker: On (switch only, no volume)
SOF Pre Mixer: 45/45 (0dB)
SOF Post Mixer: 45/45 (0dB)
SOF Deepbuffer: 45/45 (0dB)
EasyEffects: Not running, autostart disabled
UCM2: Community config minus Post Mixer override
Hotkeys: Working (increase/decrease/mute all functional)
Both speaker pairs: Active (tweeters + woofers)
```

### Forum Post
Wrote comprehensive forum post documenting the full fix. Files saved at:
- `/tmp/forum-post-legion-audio.json` — Post content (title, tags, markdown body)
- `/tmp/post-to-forum.php` — PHP script for database insertion on IX server

**Not yet published** — IX server unreachable (Tailscale stopped). To publish:
```bash
sudo tailscale up
sshpass -p 'Gptf*77ttb!@#!@#' scp -o StrictHostKeyChecking=no -o PubkeyAuthentication=no /tmp/forum-post-legion-audio.json /tmp/post-to-forum.php root@172.16.3.10:/tmp/
sshpass -p 'Gptf*77ttb!@#!@#' ssh -o StrictHostKeyChecking=no -o PubkeyAuthentication=no root@172.16.3.10 'php /tmp/post-to-forum.php'
```

Forum tags: Gadgets & Hardware (6), How-Tos & Tips (7)

## Pending/Incomplete Tasks

### Forum Post Publication
Forum post written but not published — needs Tailscale/VPN connection to IX server.

### ITE Keyboard hwdb Rule
Created `/etc/udev/hwdb.d/90-ite-keyboard-fix.hwdb` — this was part of an early debugging attempt for the volume hotkey issue. The actual fix was removing EasyEffects (not the hwdb rule). This file can be removed if it causes issues, but it's harmless.

### UCM2 Package Update Risk
The modified UCM2 config at `/usr/share/alsa/ucm2/HDA/HiFi-analog.conf` will be overwritten by `alsa-ucm-conf` package updates. Options:
1. Re-apply the edit after each update (remove the Legion Post Mixer block)
2. Pin the `alsa-ucm-conf` package version
3. Use a pacman hook to auto-patch after updates

### Future Kernel Updates
When CachyOS pushes a new kernel update, the AW88399 patch will be lost. Need to either:
1. Rebuild with the patch each time (keep PKGBUILD + patch in `/home/guru/kernel-build/`)
2. Watch for upstream inclusion (Kernel Bugzilla #218329)
3. Watch CachyOS issues #687 and #707 for native inclusion

### Cleanup
- Remove `/home/guru/kernel-build/build/` (large build artifacts)
- Keep `/home/guru/kernel-build/PKGBUILD` and `aw88399-hda-scodec.patch` for future rebuilds

### If Reboot Breaks Audio
1. Boot LTS kernel from Limine menu
2. Or boot a btrfs snapshot (available in Limine)
3. Or remove the patched kernel: `sudo pacman -S linux-cachyos` (reinstall from repo)
4. Restore UCM2 backups:
   ```bash
   sudo cp /usr/share/alsa/ucm2/HDA/HiFi-analog.conf.bak /usr/share/alsa/ucm2/HDA/HiFi-analog.conf
   sudo cp /usr/share/alsa/ucm2/HDA/HiFi-mic.conf.bak /usr/share/alsa/ucm2/HDA/HiFi-mic.conf
   ```

## Reference Information

### Key URLs
- Community fix repo: https://github.com/nadimkobeissi/16iax10h-linux-sound-saga
- CachyOS Issue #687: https://github.com/CachyOS/linux-cachyos/issues/687
- CachyOS Issue #707: https://github.com/CachyOS/linux-cachyos/issues/707
- Kernel Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=218329
- CachyOS Forum thread: https://discuss.cachyos.org/t/new-fix-for-lenovo-legion-pro-7i-16iax10h-with-alc3306-codec/18889/1
- plasma-pa virtual sink bug: https://bugs.kde.org/show_bug.cgi?id=456310

### Key File Paths
- Kernel build dir: `/home/guru/kernel-build/`
- SOF force config: `/etc/modprobe.d/sof-force.conf`
- AW88399 firmware: `/lib/firmware/aw88399_acf.bin`
- UCM2 configs: `/usr/share/alsa/ucm2/HDA/HiFi-analog.conf`, `HiFi-mic.conf`
- UCM2 backups: same path with `.bak` extension
- EasyEffects preset: `~/.local/share/easyeffects/output/Legion-Speakers-v2.json` (no longer used)
- EasyEffects autostart: `~/.config/autostart/com.github.wwmm.easyeffects.desktop`
- ITE keyboard hwdb: `/etc/udev/hwdb.d/90-ite-keyboard-fix.hwdb`
- Forum post files: `/tmp/forum-post-legion-audio.json`, `/tmp/post-to-forum.php`

### PCI/Device IDs
- PCI device: `80:1f.3` (Intel 800 Series ACE)
- PCI vendor/device: `8086:7f50`
- PCI subsystem: `17aa:3d6c`
- Codec SSID: `17aa:3906`
- ACPI amp device: `AWDZ8399:00`
- I2C amp addresses: `0x34` (right speaker), `0x35` (left speaker)
- I2C amp chip ID: `0x8321` (AW88399)
- DMI product: `83F5`
- ITE keyboard: USB VID `048d`, PID `c197`

### Speaker Layout
- **Pin 0x14** ("Speaker"): Tweeters near screen hinge, DAC 0x02, direct ALC287 output, `Speaker Playback Volume/Switch`
- **Pin 0x17** ("Bass Speaker"): Woofers near palmrest, DAC 0x06, through AW88399 I2C amps, `Bass Speaker Playback Switch` (no volume control)

### Key Technical Findings
- `PreferredDevice::sink()` in plasma-pa's audioshortcutsservice skips virtual sinks — volume up/down shortcuts silently fail while mute works fine
- SOF DSP `Post Mixer Analog Playback Volume` has coarse 2dB steps (0-45 range, -90dB to 0dB) — unsuitable as primary volume control
- HDA codec `Speaker Playback Volume` has fine 0.75dB steps (0-87 range, -65.25dB to 0dB) — but only controls tweeter DAC
- AW88399 amps go active/suspended via runtime PM — both show `stream=1, channel=0` during playback
- Both DACs (0x02 and 0x06) receive audio simultaneously from the same PCM stream
- **ALSA hardware volume only affects tweeters** — Bass Speaker (pin 0x17/DAC 0x06/AW88399) has no volume control, only on/off switch. PipeWire software volume is required for balanced scaling.

## Update: 07:15 — Volume Balance Fix, Forum Post Updates, Upstream Contributions

### Session Summary
Discovered and fixed a critical volume balance issue (Issue 3): ALSA hardware volume only controls the tweeter DAC while woofers run at fixed amp gain. Also published and iteratively refined a comprehensive forum post, drafted comments for Kernel Bugzilla, CachyOS GitHub, and the community patch repo.

### Issue 3: Tweeter/Woofer Volume Imbalance

**Discovery:** At 100% volume the speaker mix sounded balanced, but at lower volumes (~60%) sound became muffled/bass-heavy. The tweeters got quieter but woofers stayed at full volume.

**Root cause:** `Speaker Playback Volume` (the UCM2-mapped volume control) only controls DAC 0x02 (tweeters). DAC 0x06 (woofers via AW88399 amps) has NO ALSA volume control — only `Bass Speaker Playback Switch` (on/off). At 60% PipeWire volume, tweeters were at -12.75dB while woofers remained at 0dB.

**Fix applied:**
1. Removed `PlaybackMixerElem`, `PlaybackMasterElem`, and `PlaybackVolume` from the UCM2 Speaker Value block in `/usr/share/alsa/ucm2/HDA/HiFi-analog.conf`
2. This forces PipeWire to use **software volume** — digital PCM attenuation before the codec, affecting both DACs equally
3. Locked ALSA Master and Speaker at 100% (0dB) permanently
4. Created WirePlumber rule for belt-and-suspenders: `~/.config/wireplumber/wireplumber.conf.d/51-legion-speaker-volume.conf`

**Final UCM2 Speaker Value block:**
```
Value {
    PlaybackPriority 100
    PlaybackPCM "hw:${CardId}"
    PlaybackSwitch "Speaker Playback Switch"
}
```

### Configuration Changes This Update

#### Files Modified
- `/usr/share/alsa/ucm2/HDA/HiFi-analog.conf` — Removed PlaybackMixerElem/PlaybackMasterElem/PlaybackVolume from Speaker section (in addition to the earlier Post Mixer override removal)

#### Files Created
- `~/.config/wireplumber/wireplumber.conf.d/51-legion-speaker-volume.conf` — WirePlumber rule forcing `api.alsa.soft-mixer = true` on the speaker sink

### Forum Post Publication & Updates

**Published:** https://community.azcomputerguru.com/d/7-fix-lenovo-legion-pro-7-16iax10h-speaker-audio-on-linux-aw88399-smart-amps
- Discussion #7, Post #3 in Flarum database
- Tags: Gadgets & Hardware (6), How-Tos & Tips (7)

**Issue encountered:** Initial post showed "error displaying content" because raw Markdown was inserted into the database. Flarum requires content parsed through its formatter (converts Markdown to XML). Fixed by bootstrapping Flarum's PHP app and using `Formatter::parse()`.

**Flarum bootstrap pattern for future posts:**
```php
require '/home/azcomputerguru/public_html/community/vendor/autoload.php';
$basePath = '/home/azcomputerguru/public_html/community';
$paths = new \Flarum\Foundation\Paths(['base' => $basePath, 'public' => "$basePath/public", 'storage' => "$basePath/storage"]);
$configData = include "$basePath/config.php";
$config = new \Flarum\Foundation\Config($configData);
$site = new \Flarum\Foundation\InstalledSite($paths, $config);
$app = $site->bootApp();
$formatter = $app->getContainer()->make(\Flarum\Formatter\Formatter::class);
$actor = \Flarum\User\User::find(1);
$parsed = $formatter->parse($markdownContent, null, $actor);
```

**Iterative updates made to the post:**
1. Fixed rendering error (raw Markdown → Flarum XML)
2. Fixed bounty text from present tense ("There's a $2,000 bounty") to past tense ("was paid out") — bounty was claimed per Tom's Hardware article
3. Added Issue 3 (volume balance fix)
4. Added "Beyond the Community Patch" section explaining our unique findings
5. Linkified all Bugzilla, CachyOS, and KDE bug references
6. Updated heading to "Three Issues We Hit"
7. Updated Final Working Configuration to reflect software volume

**Bounty background:** The $2,000 bounty was crowdfunded by 6 people (Alderon Games $1,000, Nadim Kobeissi $500, three others $200/$200/$70/$30). It was paid out. Covered by Tom's Hardware and Hacker News.

### Upstream Contribution Drafts

Three comment files drafted and opened in text editor for manual posting:

1. **`/tmp/bugzilla-comment.txt`** → [Kernel Bugzilla #218329](https://bugzilla.kernel.org/show_bug.cgi?id=218329)
   - Detailed technical report of both UCM2 issues
   - Proposed solution with code snippets
   - System details

2. **`/tmp/cachyos-comment.txt`** → [CachyOS Issue #707](https://github.com/CachyOS/linux-cachyos/issues/707)
   - Condensed version referencing Bugzilla comment

3. **`/tmp/github-issue-nadim.txt`** → [nadimkobeissi/16iax10h-linux-sound-saga](https://github.com/nadimkobeissi/16iax10h-linux-sound-saga/issues/new)
   - Full issue with title: "UCM2 volume routing issues — tweeter/woofer imbalance and Post Mixer clarity loss"
   - Includes both issues, fix, and KDE hotkey bonus note

**Status:** Files opened in text editor for user to copy/paste manually (no `gh` CLI available, Bugzilla requires browser auth)

### Comparison: Our Fix vs Community Patch

| Aspect | Community Repo | Our Fix |
|--------|---------------|---------|
| Kernel patch | Same | Same |
| Firmware | Same | Same |
| UCM2 | Their config as-is (Post Mixer override) | Remove Post Mixer override AND Speaker volume mapping |
| Volume control | Post Mixer Analog (2dB steps, -18dB at 50%) | PipeWire software volume (both DACs scale equally) |
| Volume balance | Not addressed (masked by Post Mixer) | Fixed via software volume |
| EasyEffects | Not mentioned | Warn against (breaks KDE hotkeys) |

### Bluetooth Note
Bluetooth CAD-6005 audio adapter temporarily lost audio routing after PipeWire restarts. Not caused by our UCM2 changes — WirePlumber just reset the default sink. Fixed with `wpctl set-default 87`. Auto-switch policy is intact.

### Final Working Audio Configuration (Updated)
```
Kernel: 6.19.9-1-cachyos (patched with AW88399 HDA side-codec)
Default sink: alsa_output.pci-0000_80_1f.3-platform-skl_hda_dsp_generic.HiFi__Speaker__sink
Volume control: PipeWire SOFTWARE volume (no ALSA hardware volume mapping)
ALSA Master: Locked at 100% (0dB)
ALSA Speaker: Locked at 100% (0dB)
ALSA Bass Speaker: On (switch only, no volume)
SOF Pre Mixer: 45/45 (0dB)
SOF Post Mixer: 45/45 (0dB)
SOF Deepbuffer: 45/45 (0dB)
EasyEffects: Not running, autostart disabled
UCM2: Community config minus Post Mixer override AND minus Speaker volume mapping
WirePlumber: api.alsa.soft-mixer = true on speaker sink
Hotkeys: Working (increase/decrease/mute all functional)
Both speaker pairs: Active (tweeters + woofers scale together)
```

### All Files Created/Modified This Session (Complete List)
- `/usr/share/alsa/ucm2/HDA/HiFi-analog.conf` — Removed Legion Post Mixer block (lines 130-147) AND Speaker volume mapping (PlaybackMixerElem/MasterElem/Volume)
- `~/.config/autostart/com.github.wwmm.easyeffects.desktop` — Added `Hidden=true`
- `/etc/udev/hwdb.d/90-ite-keyboard-fix.hwdb` — ITE keyboard hwdb (harmless, from early debugging)
- `~/.config/wireplumber/wireplumber.conf.d/51-legion-speaker-volume.conf` — Soft mixer rule
- `~/.config/pipewire/pipewire.conf.d/` — Directory created (empty, filter-chain config was removed)
- `/tmp/bugzilla-comment.txt` — Kernel Bugzilla comment draft
- `/tmp/cachyos-comment.txt` — CachyOS GitHub comment draft
- `/tmp/github-issue-nadim.txt` — Community repo issue draft
- `/tmp/forum-post-legion-audio.json` — Forum post content (canonical source)
- `/tmp/post-to-forum.php` — Flarum database insertion script
- Various `/tmp/*.php` fix scripts for forum post updates

### Pending/Incomplete Tasks
1. **Post upstream comments** — User needs to manually paste from the three `/tmp/*.txt` files into Bugzilla, CachyOS GitHub, and nadimkobeissi GitHub
2. **UCM2 package update risk** — Modified `/usr/share/alsa/ucm2/HDA/HiFi-analog.conf` will be overwritten by `alsa-ucm-conf` package updates
3. **Kernel update risk** — AW88399 patch lost on kernel updates, rebuild needed
4. **Cleanup** — `/home/guru/kernel-build/build/` still contains large build artifacts

## Update: 13:45 — Post-Show Workflow, Audio Processor Tool, Voice Profiling

### Session Summary
Designed and built the post-show content workflow and radio show audio processor tool. Built a working voice profiler with validated speaker identification. GPU hit an error state during batch processing — needs reboot.

### Post-Show Workflow
Created `projects/radio-show/post-show-workflow.md` — 3-phase content pipeline:
1. **Phase 1:** Post-show debrief questionnaire (what aired vs. prep)
2. **Phase 2:** Three tiers of content generation:
   - Episode post for radio website (Astro markdown)
   - Forum discussion thread (Flarum)
   - Deep-dive blog posts (1-3 per episode, SEO-rich)
3. **Phase 3:** Cross-promotion and engagement

Identified 10 engagement gaps: no social media, newsletter placeholder, podcast distribution unclear, no SEO structured data, no audiograms, no caller follow-up, no weekly email, forum needs better seeding, no guest pipeline, no analytics-driven topics.

### Audio Processor Tool
Built `projects/radio-show/audio-processor/` — Python CLI tool with 6-stage pipeline.

**Architecture:**
- Stage 1: Transcription (faster-whisper large-v3, GPU) — **WORKING**
- Stage 2: Speaker diarization (WavLM x-vector embeddings) — **WORKING**
- Stage 3: Segment detection (multi-signal classifier) — **WORKING (basic)**
- Stage 4: Commercial removal (ffmpeg) — **CODED, untested**
- Stage 5: Segment splitting (ffmpeg) — **CODED, untested**
- Stage 6: Content analysis (Ollama) — **CODED, untested**

**Key design decisions:**
- Used WavLM (microsoft/wavlm-base-sv) instead of speechbrain ECAPA-TDNN due to torchaudio/CUDA 13.1 incompatibility
- Patched torchaudio CUDA version check (12.8 vs 13.1) — warning instead of error
- CUDA 12 libs from Ollama's bundled libraries (`/usr/local/lib/ollama/cuda_v12`) needed for faster-whisper's ctranslate2
- Added to venv activate script: `export LD_LIBRARY_PATH="/usr/local/lib/ollama/cuda_v12:${LD_LIBRARY_PATH}"`
- Element library designed as a learning system (seed known elements + discover unknown from archive)
- Multi-signal commercial detector: fingerprints + speaker ID + audio characteristics + break patterns + structural heuristics

**Python environment:**
- Venv: `/home/guru/.local/share/radio-processor/`
- Created with `--system-site-packages` (uses system python-pytorch-cuda 2.10.0)
- Key packages: faster-whisper, transformers, soundfile, librosa, scikit-learn, ollama, rich, pyyaml
- CRITICAL: Must activate with `source /home/guru/.local/share/radio-processor/bin/activate` which sets LD_LIBRARY_PATH for CUDA 12

### Voice Profiling Results

**Host profile: 180 embeddings from 9 episodes (2010-2018)**
- Model: microsoft/wavlm-base-sv (512-dim x-vector)
- Stored at: `voice-profiles/mike-swanson/embedding_0000.npy` through `embedding_0179.npy`
- Composite embedding: `voice-profiles/mike-swanson/composite.npy`

**Validated accuracy (fine-grained 3s windows on 2011-03-05 HR1):**
- Host (Mike) voice: **0.90-0.98** similarity
- Callers: **0.65-0.68**
- Produced audio/voiceovers: **0.53-0.65**
- Co-host (Ken): **0.56-0.62**
- Threshold set to **0.83** (empirically determined)

**Cross-referenced with transcript — correctly identified:**
- Show intro voiceover (produced, 0.647)
- Conan O'Brien clip played during show (0.528-0.655)
- Station break/re-intro (0.547)
- Caller asking about internet access (0.674)
- Co-host discussing iPad (0.565-0.613)

### Training Data Downloaded
**9 episodes from IX server archive (151MB total):**
- `training-data/episodes/2010-10-02-hr1.mp3` (7.3MB, 44min)
- `training-data/episodes/2011-06-04-hr1.mp3` (7.4MB)
- `training-data/episodes/2011-09-10-hr1.mp3` (11MB)
- `training-data/episodes/2014-s6e05.mp3` (9.5MB)
- `training-data/episodes/2015-s7e30.mp3` (9.0MB, 45min)
- `training-data/episodes/2016-s8e42.mp3` (19MB)
- `training-data/episodes/2017-s9e26.mp3` (48MB)
- `training-data/episodes/2018-s10e17.mp3` (21MB)
- `training-data/episodes/2018-s10e21.mp3` (21MB)

**Show production elements (training-data/elements/):**
- Bumpers: 7 files (MP3+WAV) — cities_in_dust, ET_edit, rancid_riot, stereo_mc, Warnng, white_n_nerdy
- Computer Guru Elements: 5 WAV (intro_beast, intro_kick_back, intro_or_outro, outro, SHOW INTRO)
- Corrected Elements: 5 WAV (same with corrected phone number)
- Permanent Elements: 7 WAV (az_comp_guru_spot, combos, streaming, promo_window)

### Transcription Results
**Completed (2 episodes):**
- `training-data/transcripts/2010-10-02-hr1/transcript.json` (1.2MB, 534 segments)
- `test-data/output/transcript.json` (2015-s7e30, 1.2MB, 746 segments)
- `test-data/output-hr1/transcript.json` (2011-03-05, 1070 segments)

**Failed — GPU error state (6 episodes need reboot):**
- 2011-06-04-hr1, 2014-s6e05, 2016-s8e42, 2017-s9e26, 2018-s10e17, 2018-s10e21

**Transcription speed:** ~2.5 min per 45min episode on RTX 5070 Ti (17x realtime)

### GPU Error State
After extended processing (voice profiling + fine-grained analysis + multiple transcriptions), the RTX 5070 Ti entered an error state:
- `nvidia-smi` shows ERR! across all fields
- `torch.cuda.is_available()` returns False
- GPU reset not supported on laptop GPUs
- Processes holding GPU: nvidia-powerd, Discord, Chrome
- **Fix: Reboot required**

### Forum Post Updates (continued from earlier)
- Added Issue 3 (tweeter/woofer volume balance)
- Added "Beyond the Community Patch" section
- Fixed bounty references to past tense
- Linkified all bug tracker references
- Updated Final Working Configuration for software volume

### Files Created This Update
- `projects/radio-show/post-show-workflow.md` — Full post-show content workflow
- `projects/radio-show/audio-processor/` — Complete tool with:
  - `src/cli.py` — CLI entry point (8 subcommands)
  - `src/config.py` — Config loader
  - `src/transcriber.py` — Whisper GPU transcription
  - `src/diarizer.py` — Pyannote diarization (unused, needs HF token)
  - `src/voice_profiler.py` — WavLM speaker embeddings
  - `src/segment_detector.py` — Multi-signal commercial detector
  - `src/audio_editor.py` — Commercial removal + segment splitting
  - `src/analyzer.py` — Ollama content analysis
  - `src/gpu.py` — CUDA library path setup
  - `config.yaml` — Default configuration
  - `pyproject.toml` — Package config (entry point: radio-process)
  - `training-plan.md` — Archive training strategy
  - `README.md` — Full architecture documentation
- `voice-profiles/mike-swanson/` — 180 embedding files + composite + profiles.json

### Next Session: Resume Batch Training
After reboot:
```bash
source /home/guru/.local/share/radio-processor/bin/activate
cd /home/guru/ClaudeTools/projects/radio-show/audio-processor

# Verify GPU is back
python3 -c "import torch; print(torch.cuda.is_available())"

# Transcribe remaining 6 episodes
for ep in training-data/episodes/2011-06-04-hr1.mp3 training-data/episodes/2014-s6e05.mp3 training-data/episodes/2016-s8e42.mp3 training-data/episodes/2017-s9e26.mp3 training-data/episodes/2018-s10e17.mp3 training-data/episodes/2018-s10e21.mp3; do
    name=$(basename "$ep" .mp3)
    radio-process transcribe "$ep" --output "training-data/transcripts/$name"
done
```

Then: run speaker identification across all transcribed episodes, cluster non-host voices, begin element fingerprinting.

## Update: 15:00 — Dataforth Email, GPU Debug, VWP Citrix→Hyper-V Migration, ScreenConnect

### Session Summary

Multi-task session: Dataforth email forwarding, GPU error diagnosis for voice training, and major VWP infrastructure migration (Citrix XenServer → Hyper-V). Installed ScreenConnect on VWP-FILES via PowerShell Direct.

### 1. Dataforth Email Forwarding (dataforthgit@)

**Task:** AJ (Angel Lopez) at Dataforth needs messages sent to dataforthgit@dataforth.com forwarded to him.

**Discovery:** `dataforthgit@dataforth.com` is an existing alias on the **Support** shared mailbox (`support@dataforth.com`).

**Solution:** Created inbox rule on Support mailbox via Graph API:
- **Rule:** "Forward dataforthgit@ to AJ Lopez"
- **Trigger:** recipientContains `dataforthgit@dataforth.com`
- **Action:** Forward to `alopez@dataforth.com`
- **Rule ID:** `AQAAAFO12jE=`

**Auth used:** Claude-MSP-Access multi-tenant app:
- Tenant ID: `7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584`
- App ID: `fabb3421-8b34-484b-bc17-e46de9703418`
- Client Secret: `~QJ8Q~NyQSs4OcGqHZyPrA2CVnq9KBfKiimntbMO`

### 2. GPU Error Diagnosis (RTX 5070 Ti)

**Problem:** GPU entered error state during voice training batch transcription (same issue as previous session). `nvidia-smi` shows ERR! across all fields. The GPU failed ~40 min into transcription.

**Root cause investigation:**
- `NVRM: _issueRpcLarge: rpcSendMessage failed with status 0x00000062 for fn 76!` — repeating every 100ms
- No Xid errors in dmesg — only RPC communication failures
- **Runtime D3 (fine-grained power management) is enabled** — prime suspect for GPU hang during sustained compute
- GPU is in D0 power state, video memory active
- Error first appeared at 4335 seconds after boot (~72 min)
- `torch.cuda.is_available()` returned True initially, GPU loaded model into VRAM then failed

**Fix applied (pending reboot):**
- Created `/etc/modprobe.d/nvidia-no-d3.conf`: `options nvidia NVreg_DynamicPowerManagement=0`
- Plan: After reboot, run `sudo nvidia-smi -pm 1` (persistence mode)

**Diagnostic script created:** `projects/radio-show/audio-processor/gpu_debug_transcribe.py`
- Monitors GPU temp, power, utilization, VRAM, clocks every 5 seconds
- Pre-flight CUDA health check before each episode
- 10-second cooldown between episodes
- Stops at first GPU error and logs state
- Saves logs to `gpu-debug-logs/`

**Transcription status:** Only `2010-10-02-hr1` completed. 8 episodes remaining:
- 2011-06-04-hr1, 2011-09-10-hr1, 2014-s6e05, 2015-s7e30, 2016-s8e42, 2017-s9e26, 2018-s10e17, 2018-s10e21

**After reboot commands:**
```bash
sudo nvidia-smi -pm 1
source /home/guru/.local/share/radio-processor/bin/activate
cd /home/guru/ClaudeTools/projects/radio-show/audio-processor
python3 gpu_debug_transcribe.py
```

### 3. VWP Citrix XenServer → Hyper-V Migration

#### VPN Access

**Critical:** Must `sudo tailscale down` before VWP VPN — D2TESTNAS advertises `192.168.0.0/24` for Dataforth which conflicts with VWP's same subnet.

**Starlink subnet conflict:** Starlink was on `192.168.4.0/24`, same as VPN tunnel. User changed Starlink to `10.0.3.x/16` to resolve.

**Working VPN command (split tunnel):**
```bash
sudo tailscale down
sudo openvpn --config ~/Downloads/OpenVPN-Server.ovpn --auth-user-pass /etc/openvpn/vwp-auth.txt --group nobody --daemon vwp-vpn --log /tmp/vwp-vpn.log --route-noexec
# Then manually add split routes:
sudo ip route add 172.16.9.0/24 dev tun0
sudo ip route add 192.168.0.0/24 dev tun0
sudo ip route add 192.168.3.0/24 dev tun0
```

**Key:** Must use `--route-noexec` to prevent full-tunnel `0.0.0.0/1` redirect, then manually add split routes.

#### VPN Credentials
- **Auth file:** `/etc/openvpn/vwp-auth.txt` (sysadmin / r3tr0gradE99#)
- **Remote:** 4.18.160.106:1194 TCP
- **VPN IP assigned:** 192.168.4.2 or 192.168.4.3

#### WinRM Access to Hyper-V

**Installed `pywinrm`** (`pip install --user --break-system-packages pywinrm`) for remote PowerShell via WinRM.

**WinRM enabled on VWP-HYPERV1** (user ran on console):
```powershell
Enable-PSRemoting -Force
Set-Item WSMan:\localhost\Client\TrustedHosts -Value "*" -Force
New-NetFirewallRule -DisplayName "WinRM All" -Direction Inbound -Protocol TCP -LocalPort 5985 -Action Allow
```

**Python WinRM usage:**
```python
import winrm
s = winrm.Session('http://172.16.9.184:5985/wsman', auth=('sysadmin', 'r3tr0gradE99#'), transport='ntlm')
r = s.run_ps("hostname")
print(r.std_out.decode().strip())
```

#### Hyper-V Host Status (VWP-HYPERV1)

- **Hostname:** VWP-HYPERV1
- **IP:** 172.16.9.184
- **OS:** Windows Server 2025 Standard
- **Specs:** 64 vCPUs (Xeon Platinum 8180M), 256GB RAM, PowerEdge R740
- **Disk:** 10.5TB free on C:
- **vSwitch:** "Intel(R) Ethernet 10G 4P X550/I350 rNDC - Virtual Switch" (External, NIC1 at 1Gbps)
- **Physical NICs:** NIC1 (up, 1Gbps), NIC2/3/4 (disconnected)
- **Native VLAN:** 172.16.9.x (untagged)

**Existing VMs on Hyper-V:**
| VM | State | Gen | RAM | vCPUs |
|----|-------|-----|-----|-------|
| VWP-DC1 | Running | 2 | ~7.4GB | 56 |
| VWP-FILES | Running | 2 | 2GB | 16 |

**VLAN configuration:**
- Native/untagged: 172.16.9.0/24 (VWP LAN)
- VLAN 2: 192.168.0.0/24 (OldNet)
- VLAN 99: 192.168.3.0/24 (Mgt)
- UDM trunks all VLANs, defaults to selected VLAN for untagged

#### XenServer VM Inventory (source)

| VM | OS | IP | State | vCPUs | RAM | Disk |
|----|----|----|-------|-------|-----|------|
| server 2012 R2 | Server 2012 R2 Standard | 192.168.0.19 | running | 4 | 16GB | 200GB |
| BACKUP-SRV | Server 2019 Datacenter | 192.168.0.22 | running | 2 | 15GB | 240GB |
| server 2003 | Server 2003 Enterprise SP2 | 192.168.0.20 | running | 4 | 3GB | 130GB |
| XP | Windows XP | none | running | 2 | 3GB | 40GB |
| Windows 7 (32-bit) | Windows 7 | 192.168.0.40 | halted | 2 | 4GB | 80GB |

#### Server 2012 R2 Migration (IN PROGRESS)

**VDI Export running on XenServer:**
- VDI UUID: `e65ccf95-0bc7-4530-ac91-c418e667e1de`
- VM UUID: `298da244-79b5-84ed-d6e0-694825697096`
- Export command: `xe vdi-export uuid=e65ccf95-0bc7-4530-ac91-c418e667e1de filename=/mnt/hyperv/server2012r2.vhd format=vhd`
- PID: 26610 (nohup, survives disconnects)
- Destination: `//172.16.9.184/Migration` mounted at `/mnt/hyperv` (SMBv2)
- Progress at last check: **65GB of ~200GB** (~4GB/min, ~35 min remaining)
- Transfer rate: ~4GB/min over 1Gbps link

**SMB share created on Hyper-V:**
```powershell
New-SmbShare -Name 'Migration' -Path 'C:\Migration' -FullAccess 'Everyone'
New-NetFirewallRule -DisplayName 'SMB from XenServer' -Direction Inbound -Protocol TCP -LocalPort 445 -RemoteAddress 192.168.0.0/24 -Action Allow
```

**Mount on XenServer:**
```bash
mount.cifs //172.16.9.184/Migration /mnt/hyperv -o username=sysadmin,password=r3tr0gradE99#,domain=VWP,vers=2.0
```

**Planned VM creation (after export completes):**
- Generation 1 (BIOS/MBR from XenServer)
- 4 vCPUs, 16GB RAM
- NIC on VLAN 2 (192.168.0.x)
- Attach server2012r2.vhd from C:\Migration
- Boot and install Hyper-V integration services

#### ITSvc Share (C:\Shares\ITSvc on VWP-HYPERV1)
Contains installers:
- VWP-ScreenConnect.ClientSetup.msi (27.6MB)
- VWPScreenConnect.ClientSetup.exe (19.1MB)
- Ninite installers (Chrome, Firefox, .NET, WizTree)
- ISO subfolder

### 4. ScreenConnect on VWP-FILES

**VWP-FILES VM details:**
- **Hostname:** VWP-FILES.VWP.US
- **IP:** 172.16.9.107
- **OS:** Windows Server 2019 Standard
- **Hyper-V Gen:** 2
- **RAM:** 2GB, 16 vCPUs

**PowerShell Direct credentials:** `VWP\sysadmin` / `r3tr0gradE99#`

**Installation:** MSI copied via `Copy-VMFile` (Hyper-V Guest Service Interface), installed via PowerShell Direct:
```powershell
Copy-VMFile -Name 'VWP-FILES' -SourcePath 'C:\Shares\ITSvc\VWP-ScreenConnect.ClientSetup.msi' -DestinationPath 'C:\Temp\VWP-ScreenConnect.ClientSetup.msi' -CreateFullPath -FileSource Host
Invoke-Command -VMName 'VWP-FILES' -Credential ... -ScriptBlock { Start-Process msiexec.exe -ArgumentList '/i C:\Temp\VWP-ScreenConnect.ClientSetup.msi /quiet /norestart' -Wait }
```

**Issue:** Service installed but stopped immediately — "Your host has ended the remote session." User had accidentally deleted the unit in ScreenConnect console.

**Fix:** Uninstalled (`msiexec /x ... /quiet /norestart`), reinstalled same MSI. Service now **Running**.

**Service:** `ScreenConnect Client (1912bf3444b41a08)` — connects to `instance-kgc7jt-relay.screenconnect.com:443`

### 5. Memory Saved

- `reference_dataforth_contact.md` — AJ at Dataforth, dataforthgit@ email forwarding

### Credentials Used This Session

```
### Dataforth M365 (Graph API)
- Tenant ID: 7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584
- App ID: fabb3421-8b34-484b-bc17-e46de9703418
- Client Secret: ~QJ8Q~NyQSs4OcGqHZyPrA2CVnq9KBfKiimntbMO

### VWP VPN
- Auth file: /etc/openvpn/vwp-auth.txt
- User: sysadmin / r3tr0gradE99#
- Remote: 4.18.160.106:1194 TCP

### VWP XenServer (192.168.0.104)
- SSH: root / r3tr0gradE99!
- Note: $'...' quoting for !

### VWP-HYPERV1 (172.16.9.184)
- WinRM: sysadmin / r3tr0gradE99# (NTLM)
- URL: http://172.16.9.184:5985/wsman

### VWP-DC1 (172.16.9.2)
- Domain: VWP\sysadmin / r3tr0gradE99#

### VWP-FILES (172.16.9.107)
- PowerShell Direct: VWP\sysadmin / r3tr0gradE99#

### VWP iDRAC - XenServer R720 (192.168.3.30)
- SSH: root / r3tr0gradE99#
- SSH flags: -o KexAlgorithms=+diffie-hellman-group14-sha1 -o HostKeyAlgorithms=+ssh-rsa -o Ciphers=+aes128-cbc,aes256-cbc
```

### Pending/Incomplete Tasks

1. **VDI export in progress** — Server 2012 R2 exporting from XenServer to Hyper-V, ~65GB/200GB done, PID 26610 on XenServer
2. **Create Server 2012 R2 VM on Hyper-V** — After export: Gen1, 4 vCPU, 16GB RAM, VLAN 2, attach VHD
3. **GPU debug after reboot** — Run `sudo nvidia-smi -pm 1` then `python3 gpu_debug_transcribe.py`
4. **Server 2003 data migration** — Move shares/data from 192.168.0.20 (G: drive) to VWP-FILES after 2012 R2 migration
5. **Remaining XenServer VMs** — BACKUP-SRV, server 2003, XP, Windows 7 all need migration
6. **pywinrm installed** — `pip install --user --break-system-packages pywinrm` on workstation

### Files Created/Modified This Session

- `/etc/modprobe.d/nvidia-no-d3.conf` — Disable GPU Runtime D3 power management
- `projects/radio-show/audio-processor/gpu_debug_transcribe.py` — GPU diagnostic batch transcription script
- `~/.claude/projects/-home-guru-ClaudeTools/memory/reference_dataforth_contact.md` — AJ/dataforthgit memory
- `~/.claude/projects/-home-guru-ClaudeTools/memory/MEMORY.md` — Updated index

---

## Update: 17:50 — GPU Diagnosis, Mac Handoff, CLAUDE.md Case Sensitivity Fix

### Session Summary

Deep diagnosis of RTX 5070 Ti GPU firmware crash during voice training transcription. Confirmed the issue is a known NVIDIA Blackwell GSP firmware bug. Created handoff document for Mac M4 to build native audio processor. Discovered and fixed critical config bug: `claude.md` (lowercase) was never being loaded on Linux due to case-sensitive filesystem.

### Key Decisions

1. **GPU workarounds exhausted** — Tried disabling Runtime D3, enabling persistence mode, locking GPU clocks. None prevented the GSP crash. Power management is not the cause.
2. **Offload to Mac** — Rather than cross-building, created a detailed handoff doc (`MAC_BUILD_TASK.md`) for the Mac's Claude instance to build natively for M4 hardware.
3. **CLAUDE.md fix** — Renamed `claude.md` to `CLAUDE.md` in the repo. This was the root cause of behavioral differences between Mac and Linux instances.

### GPU Diagnosis Details

**Problem:** RTX 5070 Ti enters full ERR! state after ~3-5 minutes of sustained GPU compute (Whisper transcription).

**Error:** `NVRM: _issueRpcLarge: rpcSendMessage failed with status 0x00000062 for fn 76!`

**Timeline of test run:**
- 17:03: Started transcription of `2011-06-04-hr1.mp3`
- 17:03:59: GPU loaded model, 3908MB VRAM, idle
- 17:04:14: GPU at 52C, 72W, 97% utilization, 5394MB VRAM — processing
- 17:04:59: GPU at 56C, 72W, 86% utilization — normal operation
- 17:07: nvidia-smi timed out (5s) — GPU stopped responding to queries
- 17:07:20: ctranslate2 crashed: `IndexError: vector::_M_range_check` (244 segments completed)
- 17:21-17:24: dmesg flooded with NVRM rpcSendMessage 0x00000062 errors
- GPU in full ERR! state, nvidia-smi shows ERR! across all fields

**What we tried (all ineffective):**
```bash
# Disable Runtime D3 power management
sudo bash -c 'echo on > /sys/bus/pci/devices/0000:02:00.0/power/control'

# Enable persistence mode
sudo nvidia-smi -pm 1

# Lock GPU clocks
sudo nvidia-smi -lgc 300,2100
```

**Root cause:** Known NVIDIA GSP (GPU System Processor) firmware bug on all RTX 50-series (Blackwell) GPUs under Linux. Affects drivers 580.x, 590.x, 595.x. NVIDIA internal bug #5953411 filed. Cannot disable GSP on 50-series (open kernel module required).

**Key research findings:**
- RTX 50-series requires open kernel module — proprietary driver won't work
- Cannot set `NVreg_EnableGpuFirmware=0` (only worked on 30/40-series)
- ASUS ROG Zephyrus G14 with same RTX 5070 Mobile has identical issue (NVIDIA forums)
- BIOS firmware updates helped some users
- CachyOS forum has multiple threads with same symptoms
- Temperature was fine (55C), power normal (73W), VRAM not full (5.4/12GB)

**Relevant links:**
- https://discuss.cachyos.org/t/rtx-5070-blackwell-errors/24323
- https://discuss.cachyos.org/t/rtx-5070-gpu-crashing-regularly/22786
- https://forums.developer.nvidia.com/t/rtx-5070-mobile-blackwell-gsp-timeouts-0x0000ca7d-xid-79-on-kernel-6-17-driver-580-126-ubuntu-24-04-4/360897
- https://github.com/NVIDIA/open-gpu-kernel-modules/issues/1045

### Reboot Behavior Note

Both times GPU error required reboot, warm reboot hung with spinning symbol — required hard power-off. Saved to workstation memory.

### Venv Setup for Audio Processor

Created `.venv` with `--system-site-packages` (to reuse system torch 2.10.0 and avoid re-downloading 2.5GB of CUDA packages):
```bash
cd projects/radio-show/audio-processor
python3 -m venv --system-site-packages .venv
.venv/bin/pip install --no-deps -e .
.venv/bin/pip install rich faster-whisper pyannote.audio pydub librosa scikit-learn ollama pyyaml
```

**CUDA library path issue:** `faster-whisper`'s ctranslate2 needs `libcublas.so.12` but system has CUDA 13.2. The `src/gpu.py` sets `LD_LIBRARY_PATH` at runtime, but this is too late — must be set before Python launch:
```bash
LD_LIBRARY_PATH=/usr/local/lib/ollama/cuda_v12:$LD_LIBRARY_PATH .venv/bin/python ...
```

### Mac Handoff

**File created:** `projects/radio-show/audio-processor/MAC_BUILD_TASK.md`

Comprehensive handoff document for Mac Claude instance to build a native version of the audio processor on M4 hardware. Includes:
- Full context of the GPU bug and why Mac is needed
- Architecture reference from existing code
- Key thresholds and config values
- What already works (voice profiles, 1 transcript)
- What needs doing (8 remaining episode transcriptions)
- Mac-specific build notes (CPU/MPS backend, no CUDA)
- Session log references for full context

**Mac specs** (from `.claude/machines/mikes-macbook-air.md`):
- MacBook Air M4, 10-core CPU, 16GB unified memory
- macOS 26.3.1, ClaudeTools repo cloned at `/Users/azcomputerguru/ClaudeTools`
- Ollama running with qwen3:14b, codestral:22b, nomic-embed-text
- Reachable on local network at 10.3.231.192 (SSH not enabled)

### CLAUDE.md Case Sensitivity Fix (CRITICAL)

**Root cause of behavioral differences between Mac and Linux:**

The file `.claude/claude.md` (lowercase) was committed to git. Claude Code auto-loads `CLAUDE.md` (uppercase) at startup. On macOS (case-insensitive APFS), both resolve to the same file — directives always loaded. On Linux (case-sensitive ext4/btrfs), `CLAUDE.md` doesn't exist, so **all project directives were silently missing** on the Linux workstation.

This explains:
- Asking for context already available in credentials.md
- Not following delegation rules
- Confusing client locations/networks
- Behavioral inconsistency vs the Mac

**Fix:** `git mv .claude/claude.md .claude/CLAUDE.md` — committed and pushed.

### Machine Spec Created

**File created:** `.claude/machines/acg-guru-5070.md`

Linux workstation spec matching the format the Mac uses:
- Lenovo Legion Pro 7 16IAX10H
- Intel Core Ultra 9 275HX (24 cores, 5.4 GHz)
- 32GB DDR5, RTX 5070 Ti (12GB VRAM)
- Dual 954GB NVMe (btrfs root + ext4 /home)
- CachyOS, Kernel 6.19.9-1, KDE Plasma 6.6.3 (Wayland)
- NVIDIA driver 595.45.04 (open kernel module)
- Documents GPU firmware bug and custom audio kernel

### Workstation Memory Updated

Updated `reference_workstation_setup.md` with known issue: warm reboot hangs requiring hard power-off.

### Infrastructure

- **Gitea:** git.azcomputerguru.com (HTTPS, user: mike@azcomputerguru.com, pass: Gptf*77ttb123!@#-git)
- **NVIDIA Driver:** 595.45.04 (open kernel module)
- **CUDA:** 13.2 (system), 12 libs at `/usr/local/lib/ollama/cuda_v12/`
- **PyTorch:** 2.10.0 (system-wide), CUDA 13.1

### Files Created/Modified This Update

- `projects/radio-show/audio-processor/MAC_BUILD_TASK.md` — Mac build handoff document
- `projects/radio-show/audio-processor/.venv/` — Python venv with audio processor deps
- `.claude/machines/acg-guru-5070.md` — Linux workstation machine spec
- `.claude/CLAUDE.md` — Renamed from `claude.md` (case-sensitivity fix)
- `~/.claude/projects/-home-guru-ClaudeTools/memory/reference_workstation_setup.md` — Added reboot hang note

### Pending/Next Steps

1. **Mac builds audio processor** — Pull from Gitea, read MAC_BUILD_TASK.md, build natively, transcribe 8 episodes
2. **BIOS update** — Check for ASUS BIOS update for Legion Pro 7 16IAX10H (may help GPU stability)
3. **GPU after reboot** — Currently in ERR! state, needs hard power-off to recover
4. **Verify CLAUDE.md loads** — Next session on this machine should auto-load directives correctly
5. **Enable SSH on Mac** — Would make cross-machine workflow smoother (System Settings > Sharing > Remote Login)
