claudetools/clients/instrumental-music-center/docs/2026-04-13-ticket-notes.md

IMC1 Maintenance — Ticket Notes (2026-04-11 through 2026-04-13)

Client: Instrumental Music Center (IMC) Server: IMC1 (Windows Server 2016, AD DC + AIMsi SQL host + RDS), 192.168.0.2 Dates worked: Saturday 2026-04-11, Sunday 2026-04-12, finalized Monday 2026-04-13

---

Original request

Remove RDS role from IMC1 in preparation for a planned Server 2019 upgrade.

Outcome (short version)

RDS role could not be removed — blocked by an underlying Windows component-store corruption that also affects Cumulative Update apply-on-boot. The 2019 upgrade path has to be reconsidered.

Parallel/opportunistic work completed while diagnosing: 716 GB recovered on E: (stale SQL backup cleanup), GFS retention automated, AIMsi databases moved to dedicated SSD (S:), out-of-band SSH access set up for future work.

---

Work performed — billable

1. Remote-access groundwork

• Installed OpenSSH Server on IMC1 from official GitHub release (Windows built-in Add-WindowsCapability install was a ghost — binaries never landed, also a symptom of the component-store corruption)
• Registered sshd + ssh-agent services, opened TCP/22 in the firewall
• Configured key-based auth (ed25519) via C:\ProgramData\ssh\administrators_authorized_keys with proper ACLs
• Set PowerShell as default SSH shell
• Identified & documented a routing conflict: Tailscale's pfsense-2 subnet-router was advertising 192.168.0.0/24 with lower metric than the OpenVPN tunnel, making IMC1 unreachable via VPN. Workaround: disconnect Tailscale when using IMC OpenVPN.

2. SQL backup cleanup — 716 GB freed on E:

• Inventoried E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\: 66 nightly full backups, 905 GB total, covering 2026-02-01 through 2026-04-11
• Verified the off-site Cloudberry/MSP360 backup was intact before deleting locally
• Applied GFS retention manually: kept 14 dailies + 1st-of-month (16 files, 189 GB); deleted the other 50 (716 GB)
• Noted the per-backup size dropping from ~15 GB to ~11 GB around 2026-03-28 (someone archived or cleaned source data that day; unrelated to this work)

3. Automated backup retention — going forward

• Wrote C:\Scripts\Clean-AimsiBackups.ps1 implementing the same GFS policy (14 dailies + monthlies on day-1)
• Hard safety guards: minimum of 3 newest files always kept, filename-pattern validation, per-run logs to C:\Scripts\Logs\aimsi-retention-YYYYMM.log
• Registered Windows Scheduled Task IMC AIMsi Backup Retention: runs daily 23:30 as SYSTEM, 1-hour execution limit
• Test run completed successfully; script will keep backup footprint stable automatically from now on

4. AIMsi SQL database relocation (C: → S:)

• Elevated IMC\guru to sysadmin on the AIMSQL instance via single-user-mode recovery (only Windows admins had sysadmin by default; IMC\guru wasn't in that group historically)
• Moved the three user databases from C: to the dedicated Samsung 850 PRO SSD (S:\SQL\Data\):
  • AIM.mdf — 8.6 GB — production AIMsi database
  • IMC.mdf — 9.8 GB — legacy (usage unclear; kept for safety)
  • TestConv61223.mdf — 8.8 GB — leftover from a 2023-06-12 migration test; candidate for drop
• Moved tempdb + cleaned up its orphaned files on C:
• Left system DBs (master, model, msdb) on C: — moving master requires modifying SQL Server startup parameters and the benefit is marginal vs risk
• Verified AIM client launch after the move; 4 typical concurrent users connected fine
• Disk impact: C: 322 → 278 GB used (−44 GB); S: 27 → 53 GB used (+26 GB)

5. RDS removal (parked — deeper than scoped)

Root error during Uninstall-WindowsFeature RDS-RD-Server: 0x80073701 ERROR_SXS_ASSEMBLY_MISSING.

Things we tried (in order):

1. DISM /Online /Cleanup-Image /RestoreHealth → failed Error 14 (internally E_OUTOFMEMORY 0x8007000e from an oversized 168 MB COMPONENTS registry hive — normal is 30–50 MB)
2. Same with explicit /ScratchDir → failed E_ACCESSDENIED because BITS + wuauserv were stopped
3. Started BITS/wuauserv and retried → BITS idles-and-auto-stops on Server 2016; DISM still couldn't fetch payloads
4. /Source:WIM:E:\W2016\sources\install.wim:2 /LimitAccess (install media on E:) → failed CBS_E_SOURCE_MISSING. The E:\W2016 image is RTM (14393.0); the damaged assembly is from a post-RTM Cumulative Update, so the RTM source doesn't have the right version
5. Extracted KB5075999 (Feb 2026 CU) from a local MSU and ran DISM /Online /Add-Package → staged successfully (S_OK), but on reboot the apply phase failed with HRESULT_FROM_WIN32(15010) ERROR_EVT_INVALID_EVENT_DATA at onecore\admin\wmi\events\config\manproc.cpp:733 — the ETW event manifest for provider GUID {9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0} is malformed → CBS_E_INSTALLERS_FAILED → full rollback

Decision: parked. The component-store corruption is isolated (server otherwise healthy) but it prevents both RDS removal AND CU application. It's deeper than the originally scoped RDS-removal work, and needs a planning conversation before committing more time.

6. Minor housekeeping

• Re-created the missing C:\Users\guru\Downloads folder (registry pointed there, the folder itself was gone)

---

Current state of IMC1

Area	State
AIMsi production	healthy, clients connecting
SQL instance	running; user DBs on S:, system DBs on C:
SQL backups	retained by scheduled script; 189 GB footprint (vs prior 905 GB)
C: drive	278/419 GB used (~66%) — comfortable
S: drive	53/238 GB used
RDS role	still installed (removal blocked)
OS updates	latest successful CU prior to this session; KB5075999 rolled back
SSH access	working, key-based, admin group

---

Open items / recommendations (for the next conversation)

Decision needed on the Server 2019 upgrade path. Three realistic options:

Path	Rough effort	Notes
A. Repair the component store, then retry RDS removal + in-place 2019 upgrade	2–4 hours, uncertain	Next step is to identify which KB owns the malformed provider GUID {9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}, re-register its manifest via wevtutil im, retry. If the hive is just too damaged, this path fails the same way.
B. In-place Server 2019 upgrade without fixing the corruption	2–3 hours attempt; high rollback risk	Microsoft's in-place upgrade rewrites system files wholesale; in practice it often resolves corruptions like this one, but if the upgrade itself encounters the same provider-manifest issue, we're worse off.
C. Clean Server 2019 build + AD / SQL / file share / RDS migration	6–10 hours	Lowest risk of mid-flight breakage. Most predictable. Requires a cutover window. This is what I'd recommend if the upgrade is planned anyway.

Also pending (independent of the upgrade):

• Verify the IMC database (9.8 GB) is actively being used; drop if not
• Drop TestConv61223 (8.8 GB leftover from 2023-06-12 migration test) once confirmed unused
• Disable SMB1: Set-SmbServerConfiguration -EnableSMB1Protocol $false (currently enabled — security hygiene)
• IMC2, IMC-VM in AD — last-logon 2023 and 2021 respectively, likely decommissioned; clean up if confirmed
• SERVERIMC (192.168.0.63) — its role/status is unclear from AD, worth verifying what it's doing

---

Key paths and references (for the next tech)

• SSH: ssh IMC\guru@192.168.0.2 (ed25519 key, PowerShell shell)
• Authorized keys file: C:\ProgramData\ssh\administrators_authorized_keys
• Retention script: C:\Scripts\Clean-AimsiBackups.ps1
• Retention logs: C:\Scripts\Logs\aimsi-retention-YYYYMM.log
• Scheduled task: IMC AIMsi Backup Retention (daily 23:30, SYSTEM)
• SQL instance: IMC1\AIMSQL (MSSQL15 = SQL Server 2019 Express, despite folder name)
• SQL data: S:\SQL\Data\ (user DBs); C:\Program Files\Microsoft SQL Server\MSSQL15.AIMSQL\MSSQL\DATA\ (system DBs)
• SQL backups: E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\ (yes, MSSQL14 folder hosting a MSSQL15 instance's backups — legacy)
• DISM scratch + extracted KB payload: C:\DISMScratch\
• Server 2016 install media (RTM): E:\W2016\sources\install.wim
• Full session log: clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md

Credentials

• Domain admin + AIMSQL sysadmin: IMC\guru (password handled verbally, not stored in this file)
• Local admin + SSH: same account
• sa on AIMSQL: exists and enabled; password unknown (tried one candidate, wrong, no lockout triggered)