# Session Log: 2026-04-13 — Multi-client day

Long mixed-client session. Work per client is in dedicated logs; this file is the day's index + credential stash.

## Per-client / per-project logs from today

- **IMC (Instrumental Music Center)**: `clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md` — main IMC work happened 2026-04-12 but DISM rollback chasing and the client documentation were finished today
- **Valleywide**: `clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md` — security incident
- **Dataforth**: `clients/dataforth/session-logs/2026-04-13-session.md` — API planning + Hoffman call prep

## One-line per-client summary

### IMC
- Component store corruption preventing RDS removal and 2019 upgrade
- KB5075999 `/Add-Package` staged successfully but apply-on-boot failed at ETW event manifest for provider `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}` → full rollback
- Parked the RDS removal; server otherwise healthy
- Cleaned up 716 GB of old SQL backups on E:
- Wrote `C:\Scripts\Clean-AimsiBackups.ps1` + scheduled task for GFS retention
- Moved 4 SQL DBs (AIM, IMC, TestConv61223, tempdb) from C: to S:
- Elevated `IMC\guru` to AIMSQL sysadmin via single-user recovery
- Set up SSH access on IMC1 with ed25519 key
- Created `clients/instrumental-music-center/` folder + vault entry `clients/imc/imc1.sops.yaml`

### Valleywide
- Investigating repeated `scanner` account lockouts turned up an active **brute-force attack on public RDWeb** (`VWP-QBS` at 172.16.9.169)
- User removed UDM port forward; IIS reset to drain in-flight sessions
- 30-day audit: **zero successful external logons — no breach**
- Temporarily disabled domain lockout (mistake in retrospect, was restored within ~15 min)
- Added SSH key to `VWP_ADSRVR` (192.168.0.25); double-hop to VWP-QBS works via `Invoke-Command` + explicit PSCredential
- Created `clients/valleywide/` folder + vault entry `clients/vwp/adsrvr.sops.yaml` (note: sits alongside existing `vwp/dc1.sops.yaml`; IP differs, needs reconciliation next visit)

### Dataforth
- Reviewed Swagger spec for the new datasheet API
- Confirmed OAuth2 auth_code+PKCE flow (will request `client_credentials` grant for our uploader)
- Prepared question list for John Hoffman Zoom call (batch size, rate limits, idempotency, cutover plan, PDF handling, structured-record vs raw-file push)
- Hoffman will send OAuth credentials today
- No code changes yet — waiting on creds

### Miscellaneous
- Helped user triage Neptune Exchange (tsorensen → external bounce) — user resolved on their own before I connected
- Explained Defender exclusion commands for git performance (Defender vs git interference)

## Credentials used today

> Stored here for quick recovery. Full encrypted entries in `D:\vault\` (age/SOPS).

### IMC
- **IMC1** (192.168.0.2) domain admin: `IMC\guru` / `r3tr0gradE99!`
- SSH auth: ed25519 key (`guru@DESKTOP-0O8A1RL`) in `C:\ProgramData\ssh\administrators_authorized_keys`
- `AIMSQL` sysadmin: `IMC\guru` (added 2026-04-12 via single-user recovery)
- Vault entry: `D:\vault\clients\imc\imc1.sops.yaml`

### Valleywide
- **VWP_ADSRVR** (192.168.0.25) SSH: `vwp\guru` (key auth)
- **VWP_ADSRVR / VWP-QBS** domain admin: `vwp\sysadmin` / `r3tr0gradE99#`
- SSH key in `C:\ProgramData\ssh\administrators_authorized_keys` on `VWP_ADSRVR`
- Vault entries (existing, not modified): `vwp/dc1`, `vwp/quickbooks-server-idrac`, `vwp/udm`, `vwp/xenserver`
- Vault entry (added today): `D:\vault\clients\vwp\adsrvr.sops.yaml`

### Neptune (Dataforth Exchange)
- `neptune.acghosting.com` (67.206.163.124): `ACG\administrator` / `Gptf*77ttb##`
- Access: WinRM NTLM over VPN; requires TrustedHosts on client side
- Vault: `D:\vault\clients\dataforth\neptune-exchange.sops.yaml` (existing)

### Dataforth API
- OAuth creds pending from Hoffman (expected 2026-04-13)
- Swagger's own client (not for our use): `client_id = dataforth.swagger`
- Old upload path (being retired): `DataforthWebShare` / `Data6277`

## Key commands / techniques captured

### Remote shell quirks
- `$` chars in Windows service names (e.g. `MSSQL$AIMSQL`) get eaten by bash when tunneled through SSH → PowerShell. Escape as `\$AIMSQL` in the bash-level string.
- Backticks in PowerShell here-strings can break the bash outer layer. Write to a file with `Write` and run with `powershell -File` for anything non-trivial.
- When SSH-ing into Windows OpenSSH and dispatching to a SECOND host via `Invoke-Command`, key auth doesn't carry Kerberos → need explicit PSCredential. Example:

      $pw = ConvertTo-SecureString 'r3tr0gradE99#' -AsPlainText -Force
      $cred = New-Object System.Management.Automation.PSCredential('vwp\sysadmin', $pw)
      Invoke-Command -ComputerName VWP-QBS -Credential $cred -ScriptBlock { ... }

### SQL Server single-user recovery to grant sysadmin
When Windows admin isn't already a sysadmin on an instance:

      Stop-Service 'MSSQL$AIMSQL' -Force
      Stop-Service 'MSSQLFDLauncher$AIMSQL' -Force -ErrorAction SilentlyContinue
      net start 'MSSQL$AIMSQL' /mSQLCMD
      # Connect as any local admin (granted sysadmin in -m mode):
      sqlcmd -S localhost\AIMSQL -E -Q "CREATE LOGIN [DOMAIN\user] FROM WINDOWS; ALTER SERVER ROLE sysadmin ADD MEMBER [DOMAIN\user];"
      Stop-Service 'MSSQL$AIMSQL' -Force
      Start-Service 'MSSQL$AIMSQL'
      Start-Service 'MSSQLFDLauncher$AIMSQL'

### Move SQL database files
Per user database:

      ALTER DATABASE [dbname] SET OFFLINE WITH ROLLBACK IMMEDIATE;
      ALTER DATABASE [dbname] MODIFY FILE (NAME=<logical>, FILENAME='new\path\file.mdf');
      -- physically move the file on disk
      ALTER DATABASE [dbname] SET ONLINE;

tempdb is different: `MODIFY FILE` + service restart; service recreates files at new location automatically. Delete old tempdb files from original path.

### Windows OpenSSH key auth for admin accounts
Admin-group users share one key file:

      $authFile = 'C:\ProgramData\ssh\administrators_authorized_keys'
      Set-Content -Path $authFile -Value 'ssh-ed25519 AAAA... user@host' -Encoding ASCII
      icacls $authFile /inheritance:r
      icacls $authFile /grant "Administrators:F" "SYSTEM:F"
      Restart-Service sshd

### DISM repair from a KB cab (when WU broken/blocked)
Expand MSU, then DISM /Add-Package:

      expand -f:* windows10.0-kb5075999-x64_...msu C:\DISMScratch\KB5075999
      DISM /Online /Add-Package /PackagePath:C:\DISMScratch\KB5075999\Windows10.0-KB5075999-x64.cab /ScratchDir:C:\DISMScratch

## Open / pending items

### IMC
- Decide 2019 migration path: in-place vs. clean
- Consider dropping `TestConv61223` DB (leftover from 2023-06-12 test)
- Verify `IMC` DB (9.8 GB) usage; drop if dead
- Disable SMB1 (`Set-SmbServerConfiguration -EnableSMB1Protocol $false`)

### Valleywide
- Audit UDM for UPnP (prevents the server from re-punching a hole)
- Rotate `scanner` AD account password (last set 2024-10-17)
- Investigate `LastLogonDate: 9/28/2049` ghost on VWP-QBS AD object (cosmetic)
- If RDWeb needs to go public again: IPBan + IP allowlist first
- Reconcile `vwp/adsrvr.sops.yaml` (new) vs `vwp/dc1.sops.yaml` (existing) — may be same server multi-homed, or separate DC

### Dataforth
- Await OAuth creds from Hoffman
- Store creds in `D:\vault\clients\dataforth\dataforth-api-oauth.sops.yaml` when received
- Push back for `client_credentials` grant on a dedicated uploader client
- Build POC uploader (get token → POST one file → GET + verify)
- Plan initial backfill of 501K files

## Vault changes

- Created: `D:\vault\clients\imc\imc1.sops.yaml` (encrypted)
- Created: `D:\vault\clients\vwp\adsrvr.sops.yaml` (encrypted)

## Documentation changes

- Created: `clients/instrumental-music-center/README.md`
- Created: `clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md`
- Created: `clients/valleywide/README.md`
- Created: `clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md`
- Created: `clients/dataforth/session-logs/2026-04-13-session.md`
- Created: this file