Files

Mike Swanson 5169936cfc Session log: IMC SQL move + DISM repair attempt, VWP RDWeb brute-force incident, Dataforth API planning

- IMC: document 716 GB SQL backup cleanup, retention scheduled task, DB move C:->S:, sysadmin grant via single-user recovery, parked RDS removal after KB5075999 apply rolled back on ETW manifest error
- Valleywide: document RDWeb brute-force incident on VWP-QBS, UDM port forward closure, 30-day audit showing no breach, lockout policy restoration
- Dataforth: capture Swagger API review and Hoffman Zoom call prep

2026-04-13 15:40:43 -07:00

5.2 KiB

Raw Permalink Blame History

Session Log: 2026-04-12 — IMC1 Cleanup, SSH Setup, SQL Move

Summary

Originally engaged to help remove RDS from IMC1 as prep for a Server 2019 upgrade. Removal failed with 0x80073701 (component store corruption). Spent most of the session setting up SSH access, diagnosing the corruption, performing SQL backup cleanup and DB relocation, and ultimately parking the RDS removal as a deeper problem than scoped.

Work Completed

Remote access

Installed OpenSSH Server on IMC1 via GitHub release (built-in Add-WindowsCapability install was a ghost — binaries never landed due to component store corruption)
Registered sshd and ssh-agent services, opened firewall port 22
Added public key to C:\ProgramData\ssh\administrators_authorized_keys with correct ACLs (inheritance off, Administrators + SYSTEM full control)
Set PowerShell as default SSH shell via registry
Diagnosed routing conflict: Tailscale's pfsense-2 was advertising 192.168.0.0/24 with lower metric than OpenVPN; disconnecting Tailscale restored IMC reachability

SQL backup cleanup

Inventoried E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\: 66 AIMsi nightly fulls totaling 905 GB (Feb 1 → Apr 11, 2026)
Confirmed Cloudberry off-site exists before deletion
Applied GFS retention manually: kept 14 dailies + 1st-of-month (16 files / 189 GB); deleted 50 files / 716 GB freed on E:
Noted size drop from ~15 GB → ~11 GB around 2026-03-28 suggests someone purged/archived data that day

Automated retention

Wrote C:\Scripts\Clean-AimsiBackups.ps1 implementing GFS policy
Safety: 3-newest override, filename-pattern guard, log to C:\Scripts\Logs\aimsi-retention-YYYYMM.log
Registered scheduled task IMC AIMsi Backup Retention: daily 23:30, SYSTEM, highest privileges, 1h execution limit
Test ran successfully

SQL database relocation (C: → S:)

Elevated IMC\guru to sysadmin on AIMSQL instance via single-user recovery mode (net stop → net start MSSQL$AIMSQL /mSQLCMD → ALTER SERVER ROLE sysadmin ADD MEMBER → normal restart)
Moved user databases via ALTER DATABASE ... SET OFFLINE / MODIFY FILE / SET ONLINE:
- AIM (8.6 GB)
- IMC (9.8 GB)
- TestConv61223 (8.8 GB) — still hanging on; candidate for drop
Moved tempdb via ALTER DATABASE tempdb MODIFY FILE + service restart; cleaned up orphaned files on C:
Left system DBs (master, model, msdb) on C: — moving master requires startup-parameter changes, marginal benefit
Result: C: 322→278 GB used, S: 27→53 GB used; AIM client launch tested working

Minor fix

Recreated missing C:\Users\guru\Downloads folder (registry pointed there, folder didn't exist)

RDS Removal / Component Store (parked)

Root error: 0x80073701 ERROR_SXS_ASSEMBLY_MISSING on RDS role removal.

Attempts made:

DISM /Online /Cleanup-Image /RestoreHealth — failed Error 14 (really E_OUTOFMEMORY 0x8007000e from oversized 168 MB COMPONENTS hive)
With explicit /ScratchDir — failed E_ACCESSDENIED (BITS + wuauserv were stopped; DISM couldn't fetch payloads)
Started BITS/wuauserv, retried — failed again; BITS idle-auto-stops on Server 2016 (known)
/Source:WIM:E:\W2016\sources\install.wim:2 /LimitAccess — failed CBS_E_SOURCE_MISSING (E:\W2016 is RTM 14393.0 media; damaged assembly is from a post-RTM CU)
Extracted KB5075999 (Feb 2026 CU) from local MSU at C:\Users\guru\Documents\Downloads\ → DISM /Add-Package → staged successfully (S_OK) but on reboot, apply phase failed with HRESULT_FROM_WIN32(15010) ERROR_EVT_INVALID_EVENT_DATA at onecore\admin\wmi\events\config\manproc.cpp line 733 — ETW event manifest for provider GUID {9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0} is malformed → CBS_E_INSTALLERS_FAILED → full rollback

Decision: deeper than scoped. Server otherwise healthy. RDS removal is blocking a planned 2019 upgrade.

Next actions (for next session)

Decide 2019 upgrade strategy:
- Path A: identify specific KB owning provider GUID {9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}, re-register its manifest via wevtutil im, retry CU apply
- Path B: try in-place Server 2019 upgrade despite corruption — OS files get rewritten wholesale
- Path C: clean 2019 build + AD/SQL/file/RDS migration
Verify whether IMC database (9.8 GB) is actively used; drop if not
Verify TestConv61223 can be dropped safely (leftover migration test from 2023-06-12)
Disable SMB1 (security hygiene): Set-SmbServerConfiguration -EnableSMB1Protocol $false
Add IMC entry to SOPS vault

Key Files and Paths

SSH key authorized: C:\ProgramData\ssh\administrators_authorized_keys (ed25519 guru@DESKTOP-0O8A1RL)
Retention script: C:\Scripts\Clean-AimsiBackups.ps1
Retention logs: C:\Scripts\Logs\aimsi-retention-YYYYMM.log
DISM scratch: C:\DISMScratch
Expanded KB5075999 payload: C:\DISMScratch\KB5075999\
Local Server 2016 media: E:\W2016\sources\install.wim (RTM 14393.0, index 2 = Standard Desktop Experience)

Credentials Referenced

IMC\guru — domain admin, AIMSQL sysadmin. Password handled verbally, not stored here.
sa on AIMSQL — exists, enabled, password unknown (tried one candidate, failed — no lockout policy was hit)

5.2 KiB Raw Permalink Blame History