Session log: IMC SQL move + DISM repair attempt, VWP RDWeb brute-force incident, Dataforth API planning
- IMC: document 716 GB SQL backup cleanup, retention scheduled task, DB move C:->S:, sysadmin grant via single-user recovery, parked RDS removal after KB5075999 apply rolled back on ETW manifest error - Valleywide: document RDWeb brute-force incident on VWP-QBS, UDM port forward closure, 30-day audit showing no breach, lockout policy restoration - Dataforth: capture Swagger API review and Hoffman Zoom call prep
This commit is contained in:
152
session-logs/2026-04-13-session.md
Normal file
152
session-logs/2026-04-13-session.md
Normal file
@@ -0,0 +1,152 @@
|
||||
# Session Log: 2026-04-13 — Multi-client day
|
||||
|
||||
Long mixed-client session. Work per client is in dedicated logs; this file is the day's index + credential stash.
|
||||
|
||||
## Per-client / per-project logs from today
|
||||
|
||||
- **IMC (Instrumental Music Center)**: `clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md` — main IMC work happened 2026-04-12 but DISM rollback chasing and the client documentation were finished today
|
||||
- **Valleywide**: `clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md` — security incident
|
||||
- **Dataforth**: `clients/dataforth/session-logs/2026-04-13-session.md` — API planning + Hoffman call prep
|
||||
|
||||
## One-line per-client summary
|
||||
|
||||
### IMC
|
||||
- Component store corruption preventing RDS removal and 2019 upgrade
|
||||
- KB5075999 `/Add-Package` staged successfully but apply-on-boot failed at ETW event manifest for provider `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}` → full rollback
|
||||
- Parked the RDS removal; server otherwise healthy
|
||||
- Cleaned up 716 GB of old SQL backups on E:
|
||||
- Wrote `C:\Scripts\Clean-AimsiBackups.ps1` + scheduled task for GFS retention
|
||||
- Moved 4 SQL DBs (AIM, IMC, TestConv61223, tempdb) from C: to S:
|
||||
- Elevated `IMC\guru` to AIMSQL sysadmin via single-user recovery
|
||||
- Set up SSH access on IMC1 with ed25519 key
|
||||
- Created `clients/instrumental-music-center/` folder + vault entry `clients/imc/imc1.sops.yaml`
|
||||
|
||||
### Valleywide
|
||||
- Investigating repeated `scanner` account lockouts turned up an active **brute-force attack on public RDWeb** (`VWP-QBS` at 172.16.9.169)
|
||||
- User removed UDM port forward; IIS reset to drain in-flight sessions
|
||||
- 30-day audit: **zero successful external logons — no breach**
|
||||
- Temporarily disabled domain lockout (mistake in retrospect, was restored within ~15 min)
|
||||
- Added SSH key to `VWP_ADSRVR` (192.168.0.25); double-hop to VWP-QBS works via `Invoke-Command` + explicit PSCredential
|
||||
- Created `clients/valleywide/` folder + vault entry `clients/vwp/adsrvr.sops.yaml` (note: sits alongside existing `vwp/dc1.sops.yaml`; IP differs, needs reconciliation next visit)
|
||||
|
||||
### Dataforth
|
||||
- Reviewed Swagger spec for the new datasheet API
|
||||
- Confirmed OAuth2 auth_code+PKCE flow (will request `client_credentials` grant for our uploader)
|
||||
- Prepared question list for John Hoffman Zoom call (batch size, rate limits, idempotency, cutover plan, PDF handling, structured-record vs raw-file push)
|
||||
- Hoffman will send OAuth credentials today
|
||||
- No code changes yet — waiting on creds
|
||||
|
||||
### Miscellaneous
|
||||
- Helped user triage Neptune Exchange (tsorensen → external bounce) — user resolved on their own before I connected
|
||||
- Explained Defender exclusion commands for git performance (Defender vs git interference)
|
||||
|
||||
## Credentials used today
|
||||
|
||||
> Stored here for quick recovery. Full encrypted entries in `D:\vault\` (age/SOPS).
|
||||
|
||||
### IMC
|
||||
- **IMC1** (192.168.0.2) domain admin: `IMC\guru` / `r3tr0gradE99!`
|
||||
- SSH auth: ed25519 key (`guru@DESKTOP-0O8A1RL`) in `C:\ProgramData\ssh\administrators_authorized_keys`
|
||||
- `AIMSQL` sysadmin: `IMC\guru` (added 2026-04-12 via single-user recovery)
|
||||
- Vault entry: `D:\vault\clients\imc\imc1.sops.yaml`
|
||||
|
||||
### Valleywide
|
||||
- **VWP_ADSRVR** (192.168.0.25) SSH: `vwp\guru` (key auth)
|
||||
- **VWP_ADSRVR / VWP-QBS** domain admin: `vwp\sysadmin` / `r3tr0gradE99#`
|
||||
- SSH key in `C:\ProgramData\ssh\administrators_authorized_keys` on `VWP_ADSRVR`
|
||||
- Vault entries (existing, not modified): `vwp/dc1`, `vwp/quickbooks-server-idrac`, `vwp/udm`, `vwp/xenserver`
|
||||
- Vault entry (added today): `D:\vault\clients\vwp\adsrvr.sops.yaml`
|
||||
|
||||
### Neptune (Dataforth Exchange)
|
||||
- `neptune.acghosting.com` (67.206.163.124): `ACG\administrator` / `Gptf*77ttb##`
|
||||
- Access: WinRM NTLM over VPN; requires TrustedHosts on client side
|
||||
- Vault: `D:\vault\clients\dataforth\neptune-exchange.sops.yaml` (existing)
|
||||
|
||||
### Dataforth API
|
||||
- OAuth creds pending from Hoffman (expected 2026-04-13)
|
||||
- Swagger's own client (not for our use): `client_id = dataforth.swagger`
|
||||
- Old upload path (being retired): `DataforthWebShare` / `Data6277`
|
||||
|
||||
## Key commands / techniques captured
|
||||
|
||||
### Remote shell quirks
|
||||
- `$` chars in Windows service names (e.g. `MSSQL$AIMSQL`) get eaten by bash when tunneled through SSH → PowerShell. Escape as `\$AIMSQL` in the bash-level string.
|
||||
- Backticks in PowerShell here-strings can break the bash outer layer. Write to a file with `Write` and run with `powershell -File` for anything non-trivial.
|
||||
- When SSH-ing into Windows OpenSSH and dispatching to a SECOND host via `Invoke-Command`, key auth doesn't carry Kerberos → need explicit PSCredential. Example:
|
||||
|
||||
$pw = ConvertTo-SecureString 'r3tr0gradE99#' -AsPlainText -Force
|
||||
$cred = New-Object System.Management.Automation.PSCredential('vwp\sysadmin', $pw)
|
||||
Invoke-Command -ComputerName VWP-QBS -Credential $cred -ScriptBlock { ... }
|
||||
|
||||
### SQL Server single-user recovery to grant sysadmin
|
||||
When Windows admin isn't already a sysadmin on an instance:
|
||||
|
||||
Stop-Service 'MSSQL$AIMSQL' -Force
|
||||
Stop-Service 'MSSQLFDLauncher$AIMSQL' -Force -ErrorAction SilentlyContinue
|
||||
net start 'MSSQL$AIMSQL' /mSQLCMD
|
||||
# Connect as any local admin (granted sysadmin in -m mode):
|
||||
sqlcmd -S localhost\AIMSQL -E -Q "CREATE LOGIN [DOMAIN\user] FROM WINDOWS; ALTER SERVER ROLE sysadmin ADD MEMBER [DOMAIN\user];"
|
||||
Stop-Service 'MSSQL$AIMSQL' -Force
|
||||
Start-Service 'MSSQL$AIMSQL'
|
||||
Start-Service 'MSSQLFDLauncher$AIMSQL'
|
||||
|
||||
### Move SQL database files
|
||||
Per user database:
|
||||
|
||||
ALTER DATABASE [dbname] SET OFFLINE WITH ROLLBACK IMMEDIATE;
|
||||
ALTER DATABASE [dbname] MODIFY FILE (NAME=<logical>, FILENAME='new\path\file.mdf');
|
||||
-- physically move the file on disk
|
||||
ALTER DATABASE [dbname] SET ONLINE;
|
||||
|
||||
tempdb is different: `MODIFY FILE` + service restart; service recreates files at new location automatically. Delete old tempdb files from original path.
|
||||
|
||||
### Windows OpenSSH key auth for admin accounts
|
||||
Admin-group users share one key file:
|
||||
|
||||
$authFile = 'C:\ProgramData\ssh\administrators_authorized_keys'
|
||||
Set-Content -Path $authFile -Value 'ssh-ed25519 AAAA... user@host' -Encoding ASCII
|
||||
icacls $authFile /inheritance:r
|
||||
icacls $authFile /grant "Administrators:F" "SYSTEM:F"
|
||||
Restart-Service sshd
|
||||
|
||||
### DISM repair from a KB cab (when WU broken/blocked)
|
||||
Expand MSU, then DISM /Add-Package:
|
||||
|
||||
expand -f:* windows10.0-kb5075999-x64_...msu C:\DISMScratch\KB5075999
|
||||
DISM /Online /Add-Package /PackagePath:C:\DISMScratch\KB5075999\Windows10.0-KB5075999-x64.cab /ScratchDir:C:\DISMScratch
|
||||
|
||||
## Open / pending items
|
||||
|
||||
### IMC
|
||||
- Decide 2019 migration path: in-place vs. clean
|
||||
- Consider dropping `TestConv61223` DB (leftover from 2023-06-12 test)
|
||||
- Verify `IMC` DB (9.8 GB) usage; drop if dead
|
||||
- Disable SMB1 (`Set-SmbServerConfiguration -EnableSMB1Protocol $false`)
|
||||
|
||||
### Valleywide
|
||||
- Audit UDM for UPnP (prevents the server from re-punching a hole)
|
||||
- Rotate `scanner` AD account password (last set 2024-10-17)
|
||||
- Investigate `LastLogonDate: 9/28/2049` ghost on VWP-QBS AD object (cosmetic)
|
||||
- If RDWeb needs to go public again: IPBan + IP allowlist first
|
||||
- Reconcile `vwp/adsrvr.sops.yaml` (new) vs `vwp/dc1.sops.yaml` (existing) — may be same server multi-homed, or separate DC
|
||||
|
||||
### Dataforth
|
||||
- Await OAuth creds from Hoffman
|
||||
- Store creds in `D:\vault\clients\dataforth\dataforth-api-oauth.sops.yaml` when received
|
||||
- Push back for `client_credentials` grant on a dedicated uploader client
|
||||
- Build POC uploader (get token → POST one file → GET + verify)
|
||||
- Plan initial backfill of 501K files
|
||||
|
||||
## Vault changes
|
||||
|
||||
- Created: `D:\vault\clients\imc\imc1.sops.yaml` (encrypted)
|
||||
- Created: `D:\vault\clients\vwp\adsrvr.sops.yaml` (encrypted)
|
||||
|
||||
## Documentation changes
|
||||
|
||||
- Created: `clients/instrumental-music-center/README.md`
|
||||
- Created: `clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md`
|
||||
- Created: `clients/valleywide/README.md`
|
||||
- Created: `clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md`
|
||||
- Created: `clients/dataforth/session-logs/2026-04-13-session.md`
|
||||
- Created: this file
|
||||
Reference in New Issue
Block a user