Mike Swanson
eb183512c0
- IMC: document 716 GB SQL backup cleanup, retention scheduled task, DB move C:->S:, sysadmin grant via single-user recovery, parked RDS removal after KB5075999 apply rolled back on ETW manifest error - Valleywide: document RDWeb brute-force incident on VWP-QBS, UDM port forward closure, 30-day audit showing no breach, lockout policy restoration - Dataforth: capture Swagger API review and Hoffman Zoom call prep
78 lines
5.2 KiB
Markdown
78 lines
5.2 KiB
Markdown
# Session Log: 2026-04-12 — IMC1 Cleanup, SSH Setup, SQL Move
|
|
|
|
## Summary
|
|
|
|
Originally engaged to help remove RDS from IMC1 as prep for a Server 2019 upgrade. Removal failed with `0x80073701` (component store corruption). Spent most of the session setting up SSH access, diagnosing the corruption, performing SQL backup cleanup and DB relocation, and ultimately parking the RDS removal as a deeper problem than scoped.
|
|
|
|
## Work Completed
|
|
|
|
### Remote access
|
|
- Installed OpenSSH Server on IMC1 via GitHub release (built-in `Add-WindowsCapability` install was a ghost — binaries never landed due to component store corruption)
|
|
- Registered `sshd` and `ssh-agent` services, opened firewall port 22
|
|
- Added public key to `C:\ProgramData\ssh\administrators_authorized_keys` with correct ACLs (inheritance off, Administrators + SYSTEM full control)
|
|
- Set PowerShell as default SSH shell via registry
|
|
- Diagnosed routing conflict: Tailscale's `pfsense-2` was advertising `192.168.0.0/24` with lower metric than OpenVPN; disconnecting Tailscale restored IMC reachability
|
|
|
|
### SQL backup cleanup
|
|
- Inventoried `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\`: 66 AIMsi nightly fulls totaling **905 GB** (Feb 1 → Apr 11, 2026)
|
|
- Confirmed Cloudberry off-site exists before deletion
|
|
- Applied GFS retention manually: kept 14 dailies + 1st-of-month (16 files / 189 GB); deleted 50 files / **716 GB freed on E:**
|
|
- Noted size drop from ~15 GB → ~11 GB around 2026-03-28 suggests someone purged/archived data that day
|
|
|
|
### Automated retention
|
|
- Wrote `C:\Scripts\Clean-AimsiBackups.ps1` implementing GFS policy
|
|
- Safety: 3-newest override, filename-pattern guard, log to `C:\Scripts\Logs\aimsi-retention-YYYYMM.log`
|
|
- Registered scheduled task `IMC AIMsi Backup Retention`: daily 23:30, SYSTEM, highest privileges, 1h execution limit
|
|
- Test ran successfully
|
|
|
|
### SQL database relocation (C: → S:)
|
|
- Elevated `IMC\guru` to sysadmin on `AIMSQL` instance via single-user recovery mode (net stop → `net start MSSQL$AIMSQL /mSQLCMD` → `ALTER SERVER ROLE sysadmin ADD MEMBER` → normal restart)
|
|
- Moved user databases via `ALTER DATABASE ... SET OFFLINE / MODIFY FILE / SET ONLINE`:
|
|
- `AIM` (8.6 GB)
|
|
- `IMC` (9.8 GB)
|
|
- `TestConv61223` (8.8 GB) — still hanging on; candidate for drop
|
|
- Moved `tempdb` via `ALTER DATABASE tempdb MODIFY FILE` + service restart; cleaned up orphaned files on C:
|
|
- Left system DBs (master, model, msdb) on C: — moving `master` requires startup-parameter changes, marginal benefit
|
|
- **Result:** C: 322→278 GB used, S: 27→53 GB used; AIM client launch tested working
|
|
|
|
### Minor fix
|
|
- Recreated missing `C:\Users\guru\Downloads` folder (registry pointed there, folder didn't exist)
|
|
|
|
## RDS Removal / Component Store (parked)
|
|
|
|
Root error: `0x80073701 ERROR_SXS_ASSEMBLY_MISSING` on RDS role removal.
|
|
|
|
Attempts made:
|
|
1. `DISM /Online /Cleanup-Image /RestoreHealth` — failed Error 14 (really `E_OUTOFMEMORY 0x8007000e` from oversized 168 MB COMPONENTS hive)
|
|
2. With explicit `/ScratchDir` — failed `E_ACCESSDENIED` (BITS + wuauserv were stopped; DISM couldn't fetch payloads)
|
|
3. Started BITS/wuauserv, retried — failed again; BITS idle-auto-stops on Server 2016 (known)
|
|
4. `/Source:WIM:E:\W2016\sources\install.wim:2 /LimitAccess` — failed `CBS_E_SOURCE_MISSING` (E:\W2016 is RTM 14393.0 media; damaged assembly is from a post-RTM CU)
|
|
5. Extracted KB5075999 (Feb 2026 CU) from local MSU at `C:\Users\guru\Documents\Downloads\` → `DISM /Add-Package` → **staged successfully (S_OK)** but on reboot, apply phase failed with `HRESULT_FROM_WIN32(15010) ERROR_EVT_INVALID_EVENT_DATA` at `onecore\admin\wmi\events\config\manproc.cpp line 733` — ETW event manifest for provider GUID `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}` is malformed → `CBS_E_INSTALLERS_FAILED` → full rollback
|
|
|
|
Decision: deeper than scoped. Server otherwise healthy. RDS removal is blocking a planned 2019 upgrade.
|
|
|
|
## Next actions (for next session)
|
|
|
|
- **Decide 2019 upgrade strategy:**
|
|
- Path A: identify specific KB owning provider GUID `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}`, re-register its manifest via `wevtutil im`, retry CU apply
|
|
- Path B: try in-place Server 2019 upgrade despite corruption — OS files get rewritten wholesale
|
|
- Path C: clean 2019 build + AD/SQL/file/RDS migration
|
|
- Verify whether `IMC` database (9.8 GB) is actively used; drop if not
|
|
- Verify `TestConv61223` can be dropped safely (leftover migration test from 2023-06-12)
|
|
- Disable SMB1 (security hygiene): `Set-SmbServerConfiguration -EnableSMB1Protocol $false`
|
|
- Add IMC entry to SOPS vault
|
|
|
|
## Key Files and Paths
|
|
|
|
- SSH key authorized: `C:\ProgramData\ssh\administrators_authorized_keys` (ed25519 `guru@DESKTOP-0O8A1RL`)
|
|
- Retention script: `C:\Scripts\Clean-AimsiBackups.ps1`
|
|
- Retention logs: `C:\Scripts\Logs\aimsi-retention-YYYYMM.log`
|
|
- DISM scratch: `C:\DISMScratch`
|
|
- Expanded KB5075999 payload: `C:\DISMScratch\KB5075999\`
|
|
- Local Server 2016 media: `E:\W2016\sources\install.wim` (RTM 14393.0, index 2 = Standard Desktop Experience)
|
|
|
|
## Credentials Referenced
|
|
|
|
- `IMC\guru` — domain admin, AIMSQL sysadmin. Password handled verbally, not stored here.
|
|
- `sa` on `AIMSQL` — exists, enabled, password unknown (tried one candidate, failed — no lockout policy was hit)
|