claudetools/session-logs/2026-04-13-session.md

Session Log: 2026-04-13 — Multi-client day

Long mixed-client session. Work per client is in dedicated logs; this file is the day's index + credential stash.

Per-client / per-project logs from today

• IMC (Instrumental Music Center): clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md — main IMC work happened 2026-04-12 but DISM rollback chasing and the client documentation were finished today
• Valleywide: clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md — security incident
• Dataforth: clients/dataforth/session-logs/2026-04-13-session.md — API planning + Hoffman call prep

One-line per-client summary

IMC

• Component store corruption preventing RDS removal and 2019 upgrade
• KB5075999 /Add-Package staged successfully but apply-on-boot failed at ETW event manifest for provider {9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0} → full rollback
• Parked the RDS removal; server otherwise healthy
• Cleaned up 716 GB of old SQL backups on E:
• Wrote C:\Scripts\Clean-AimsiBackups.ps1 + scheduled task for GFS retention
• Moved 4 SQL DBs (AIM, IMC, TestConv61223, tempdb) from C: to S:
• Elevated IMC\guru to AIMSQL sysadmin via single-user recovery
• Set up SSH access on IMC1 with ed25519 key
• Created clients/instrumental-music-center/ folder + vault entry clients/imc/imc1.sops.yaml

Valleywide

• Investigating repeated scanner account lockouts turned up an active brute-force attack on public RDWeb (VWP-QBS at 172.16.9.169)
• User removed UDM port forward; IIS reset to drain in-flight sessions
• 30-day audit: zero successful external logons — no breach
• Temporarily disabled domain lockout (mistake in retrospect, was restored within ~15 min)
• Added SSH key to VWP_ADSRVR (192.168.0.25); double-hop to VWP-QBS works via Invoke-Command + explicit PSCredential
• Created clients/valleywide/ folder + vault entry clients/vwp/adsrvr.sops.yaml (note: sits alongside existing vwp/dc1.sops.yaml; IP differs, needs reconciliation next visit)

Dataforth

• Reviewed Swagger spec for the new datasheet API
• Confirmed OAuth2 auth_code+PKCE flow (will request client_credentials grant for our uploader)
• Prepared question list for John Hoffman Zoom call (batch size, rate limits, idempotency, cutover plan, PDF handling, structured-record vs raw-file push)
• Hoffman will send OAuth credentials today
• No code changes yet — waiting on creds

Miscellaneous

• Helped user triage Neptune Exchange (tsorensen → external bounce) — user resolved on their own before I connected
• Explained Defender exclusion commands for git performance (Defender vs git interference)

Credentials used today

Stored here for quick recovery. Full encrypted entries in D:\vault\ (age/SOPS).

IMC

• IMC1 (192.168.0.2) domain admin: IMC\guru / r3tr0gradE99!
• SSH auth: ed25519 key (guru@DESKTOP-0O8A1RL) in C:\ProgramData\ssh\administrators_authorized_keys
• AIMSQL sysadmin: IMC\guru (added 2026-04-12 via single-user recovery)
• Vault entry: D:\vault\clients\imc\imc1.sops.yaml

Valleywide

• VWP_ADSRVR (192.168.0.25) SSH: vwp\guru (key auth)
• VWP_ADSRVR / VWP-QBS domain admin: vwp\sysadmin / r3tr0gradE99#
• SSH key in C:\ProgramData\ssh\administrators_authorized_keys on VWP_ADSRVR
• Vault entries (existing, not modified): vwp/dc1, vwp/quickbooks-server-idrac, vwp/udm, vwp/xenserver
• Vault entry (added today): D:\vault\clients\vwp\adsrvr.sops.yaml

Neptune (Dataforth Exchange)

• neptune.acghosting.com (67.206.163.124): ACG\administrator / Gptf*77ttb##
• Access: WinRM NTLM over VPN; requires TrustedHosts on client side
• Vault: D:\vault\clients\dataforth\neptune-exchange.sops.yaml (existing)

Dataforth API

• OAuth creds pending from Hoffman (expected 2026-04-13)
• Swagger's own client (not for our use): client_id = dataforth.swagger
• Old upload path (being retired): DataforthWebShare / Data6277

Key commands / techniques captured

Remote shell quirks

• $ chars in Windows service names (e.g. MSSQL$AIMSQL) get eaten by bash when tunneled through SSH → PowerShell. Escape as \$AIMSQL in the bash-level string.

• Backticks in PowerShell here-strings can break the bash outer layer. Write to a file with Write and run with powershell -File for anything non-trivial.

• When SSH-ing into Windows OpenSSH and dispatching to a SECOND host via Invoke-Command, key auth doesn't carry Kerberos → need explicit PSCredential. Example:

  $pw = ConvertTo-SecureString 'r3tr0gradE99#' -AsPlainText -Force
  $cred = New-Object System.Management.Automation.PSCredential('vwp\sysadmin', $pw)
  Invoke-Command -ComputerName VWP-QBS -Credential $cred -ScriptBlock { ... }
  

SQL Server single-user recovery to grant sysadmin

When Windows admin isn't already a sysadmin on an instance:

  Stop-Service 'MSSQL$AIMSQL' -Force
  Stop-Service 'MSSQLFDLauncher$AIMSQL' -Force -ErrorAction SilentlyContinue
  net start 'MSSQL$AIMSQL' /mSQLCMD
  # Connect as any local admin (granted sysadmin in -m mode):
  sqlcmd -S localhost\AIMSQL -E -Q "CREATE LOGIN [DOMAIN\user] FROM WINDOWS; ALTER SERVER ROLE sysadmin ADD MEMBER [DOMAIN\user];"
  Stop-Service 'MSSQL$AIMSQL' -Force
  Start-Service 'MSSQL$AIMSQL'
  Start-Service 'MSSQLFDLauncher$AIMSQL'

Move SQL database files

Per user database:

  ALTER DATABASE [dbname] SET OFFLINE WITH ROLLBACK IMMEDIATE;
  ALTER DATABASE [dbname] MODIFY FILE (NAME=<logical>, FILENAME='new\path\file.mdf');
  -- physically move the file on disk
  ALTER DATABASE [dbname] SET ONLINE;

tempdb is different: MODIFY FILE + service restart; service recreates files at new location automatically. Delete old tempdb files from original path.

Windows OpenSSH key auth for admin accounts

Admin-group users share one key file:

  $authFile = 'C:\ProgramData\ssh\administrators_authorized_keys'
  Set-Content -Path $authFile -Value 'ssh-ed25519 AAAA... user@host' -Encoding ASCII
  icacls $authFile /inheritance:r
  icacls $authFile /grant "Administrators:F" "SYSTEM:F"
  Restart-Service sshd

DISM repair from a KB cab (when WU broken/blocked)

Expand MSU, then DISM /Add-Package:

  expand -f:* windows10.0-kb5075999-x64_...msu C:\DISMScratch\KB5075999
  DISM /Online /Add-Package /PackagePath:C:\DISMScratch\KB5075999\Windows10.0-KB5075999-x64.cab /ScratchDir:C:\DISMScratch

Open / pending items

IMC

• Decide 2019 migration path: in-place vs. clean
• Consider dropping TestConv61223 DB (leftover from 2023-06-12 test)
• Verify IMC DB (9.8 GB) usage; drop if dead
• Disable SMB1 (Set-SmbServerConfiguration -EnableSMB1Protocol $false)

Valleywide

• Audit UDM for UPnP (prevents the server from re-punching a hole)
• Rotate scanner AD account password (last set 2024-10-17)
• Investigate LastLogonDate: 9/28/2049 ghost on VWP-QBS AD object (cosmetic)
• If RDWeb needs to go public again: IPBan + IP allowlist first
• Reconcile vwp/adsrvr.sops.yaml (new) vs vwp/dc1.sops.yaml (existing) — may be same server multi-homed, or separate DC

Dataforth

• Await OAuth creds from Hoffman
• Store creds in D:\vault\clients\dataforth\dataforth-api-oauth.sops.yaml when received
• Push back for client_credentials grant on a dedicated uploader client
• Build POC uploader (get token → POST one file → GET + verify)
• Plan initial backfill of 501K files

Vault changes

• Created: D:\vault\clients\imc\imc1.sops.yaml (encrypted)
• Created: D:\vault\clients\vwp\adsrvr.sops.yaml (encrypted)

Documentation changes

• Created: clients/instrumental-music-center/README.md
• Created: clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md
• Created: clients/valleywide/README.md
• Created: clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md
• Created: clients/dataforth/session-logs/2026-04-13-session.md
• Created: this file