From 5169936cfc678a10bc8d54381d5d3b55743d4f94 Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Mon, 13 Apr 2026 15:40:43 -0700 Subject: [PATCH] Session log: IMC SQL move + DISM repair attempt, VWP RDWeb brute-force incident, Dataforth API planning - IMC: document 716 GB SQL backup cleanup, retention scheduled task, DB move C:->S:, sysadmin grant via single-user recovery, parked RDS removal after KB5075999 apply rolled back on ETW manifest error - Valleywide: document RDWeb brute-force incident on VWP-QBS, UDM port forward closure, 30-day audit showing no breach, lockout policy restoration - Dataforth: capture Swagger API review and Hoffman Zoom call prep --- .../session-logs/2026-04-13-session.md | 119 ++++++++++++++ clients/instrumental-music-center/README.md | 59 +++++++ .../2026-04-12-imc1-cleanup-and-sql-move.md | 77 +++++++++ clients/valleywide/README.md | 52 ++++++ .../2026-04-13-rdweb-brute-force-incident.md | 59 +++++++ session-logs/2026-04-13-session.md | 152 ++++++++++++++++++ 6 files changed, 518 insertions(+) create mode 100644 clients/dataforth/session-logs/2026-04-13-session.md create mode 100644 clients/instrumental-music-center/README.md create mode 100644 clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md create mode 100644 clients/valleywide/README.md create mode 100644 clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md create mode 100644 session-logs/2026-04-13-session.md diff --git a/clients/dataforth/session-logs/2026-04-13-session.md b/clients/dataforth/session-logs/2026-04-13-session.md new file mode 100644 index 0000000..3290dc9 --- /dev/null +++ b/clients/dataforth/session-logs/2026-04-13-session.md @@ -0,0 +1,119 @@ +# Session Log: 2026-04-13 — Dataforth + +## Summary + +Continuation of the test datasheet pipeline work. Prior session (2026-04-12) confirmed PostgreSQL migration complete; Hoffman provided the new Swagger API URL; awaiting OAuth credentials. Today: reviewed the full API spec, prepared a structured question list for a Zoom call with Hoffman, and discussed architecture options (raw file upload vs. structured record push vs. direct DB). + +Also helped user triage an unrelated Neptune Exchange mail-flow issue (tsorensen → external bounce). User resolved on their own before I got into it. + +## Work completed + +### API spec review +Pulled `https://www.dataforth.com/swagger/v1/swagger.json` and mapped endpoints. + +**Base URL:** `https://www.dataforth.com` (presumed; Swagger UI at `/swagger/index.html`) + +**Authentication (IdentityServer-style)** +- Flow: **OAuth2 Authorization Code + PKCE** +- Authorization URL: `https://login.dataforth.com/connect/authorize` +- Token URL: `https://login.dataforth.com/connect/token` +- Scopes: `openid`, `profile`, `dataforth.web` +- Swagger's own test client: `client_id = dataforth.swagger` (NOT for our use) +- OIDC discovery expected at: `https://login.dataforth.com/.well-known/openid-configuration` + +**All endpoints** +| Path | Method | +|------|--------| +| `/api/v1/Admin/refresh-cache` | POST | +| `/api/v1/Admin/cache-status` | GET | +| `/api/v1/Categories` | GET | +| `/api/v1/Categories/{id}` | GET | +| `/api/v1/Categories/by-catalog-node/{catalogNodeId}` | GET | +| `/api/v1/OrderableProducts/{orderableProductId}/Attributes` | POST | +| `/api/v1/OrderableProducts/{orderableProductId}/Attributes/{attributeId}` | PUT/DELETE | +| `/api/v1/Products`, `/{id}`, `/by-part-number/{partNumber}` | GET | +| `/api/v1/product-series`, `/{id}`, `/by-designation/{designation}`, `/by-catalog-node/{catalogNodeId}` | GET | +| `/api/v1/ProductType`, `/{productTypeId}/products` | GET | +| `/api/v1/TestReportDataFiles` | POST (single upload) | +| `/api/v1/TestReportDataFiles` | GET (paginated list) | +| `/api/v1/TestReportDataFiles/bulk` | POST (batch upload) | +| `/api/v1/TestReportDataFiles/{serialNumber}` | GET / DELETE | +| `/api/v1/TestReportDataFiles/stats` | GET | + +**TestReportDataFiles payload shapes** +- POST single: `{ SerialNumber: string(max 50), Content: string(min 1) }` → `{ SerialNumber, ContentHash, Created }` +- POST bulk: `{ Items: [CreateTestReportRequest, ...] }` → `{ TotalReceived, Created, Updated, Unchanged, Errors[] }` +- GET single: `{ SerialNumber, Content, CreatedAtUtc, UpdatedAtUtc }` +- GET stats: `{ TotalCount, LatestCreatedAtUtc, LatestUpdatedAtUtc }` +- Server handles dedup via ContentHash → client doesn't need to pre-check. + +### Architecture discussion +Three options for delivering datasheets: +- **A: Raw file blob via current API** — works today, zero new API work, simple client code +- **B: Structured records via new endpoints** — cleaner long-term; we already have parsed data in AD2's PostgreSQL `TestDataDB` (2.8M records post-2026-04-12 migration). Requires Hoffman to add endpoints +- **C: Direct DB access** — rejected (coupling, security, DBA nightmare) + +Preferred path: whichever is less work for Hoffman. Frame it as offering flexibility — we can send raw text, structured JSON, or even CSV. + +### Questions prepared for John Hoffman Zoom call +Produced a prioritized list (MUST / SHOULD / NICE) covering: +- Batch size + payload size + rate limits (MUST) +- Idempotency + dedup semantics (MUST) +- Cutover plan from old DataforthWebShare path (MUST) +- Request: enable `client_credentials` grant on a new client for the AD2 uploader (SHOULD) +- Staging endpoint availability (SHOULD) +- PDF handling (`X:\For_Web_PDF`) — same endpoint or different? (SHOULD) +- Product linkage — does a TestReport need to link to a Product/Series record? (SHOULD) +- Monitoring + error visibility on his side (NICE) +- SLA / escalation contact (NICE) + +### Pending from Hoffman (as of end-of-session 2026-04-13) +- OAuth credentials (he said "today") +- Clarification on client_credentials grant support +- Answers to the MUST questions above after the Zoom + +## Pipeline context (unchanged from 2026-04-12) + +### Current state +- **Stage 1**: DOS test stations → D2TESTNAS (192.168.0.9, rsync daemon, module "test" → /data/test) ✓ +- **Stage 2**: NAS → AD2 via `Sync-FromNAS-rsync.ps1` scheduled every 15 min ✓ +- **Stage 3**: DFWDS.exe validates + renames — **config wiped in crypto attack**; `C:\DFWDS\DFWDS_NAMES.TXT` missing. Check Haubner D: for backup. +- **Stage 4**: Website upload — **BROKEN**; this is what we're rebuilding via the new API +- **Stage 5**: PDF generation — ~4,773 PDFs in `X:\For_Web_PDF`, origin unclear + +### Data locations +- Incoming: `X:\Test_Datasheets` (staging) +- Validated: `X:\For_Web` (~501K files) ← uploader source +- PDFs: `X:\For_Web_PDF` (~4.7K files) +- Rejected: `X:\Bad_Datasheets` (~18K) +- DFWDS logs: `X:\Datasheets_Log` +- `X:` = `\\ad2\webshare` + +### Datasheet format +Plain text, ~50 lines. Header: Dataforth address/phone. Fields: Date, Model (e.g. SCM5B41-03), SN (e.g. 178439-1), accuracy test table, final test results. Filename: `{SN}.txt` (e.g. `178439-1.txt`). + +### Credentials used/referenced +- **Old upload path** (being replaced): `DataforthWebShare / Data6277` +- **New API**: OAuth client credentials pending from Hoffman +- **Neptune Exchange** (for today's mail triage): `ACG\administrator` / `Gptf*77ttb##` — requires VPN + +## Next session plan + +1. Receive OAuth creds from Hoffman (client_id + client_secret, ideally client_credentials grant enabled) +2. Store credentials in `D:\vault\clients\dataforth\dataforth-api-oauth.sops.yaml` +3. Stand up a one-page POC: get token, POST one test report, verify via GET +4. If POC works → implement full uploader on AD2: + - Language: PowerShell (fits existing scripts) or Python (already used in `projects/dataforth-dos/datasheet-pipeline/implementation/`) + - State tracking: local manifest (serial → hash + last-upload-time) or use server's ContentHash response + - Use `/bulk` endpoint in batches (size TBD with Hoffman) + - Scheduled task on AD2, 15-min or hourly cadence + - Initial backfill script for 501K files — run off-hours +5. Parallel-run with old webshare path until confident, then retire old path + +## Reference URLs + +- Swagger UI: https://www.dataforth.com/swagger/index.html +- Swagger JSON: https://www.dataforth.com/swagger/v1/swagger.json +- Authorization URL: https://login.dataforth.com/connect/authorize +- Token URL: https://login.dataforth.com/connect/token +- Expected OIDC discovery: https://login.dataforth.com/.well-known/openid-configuration diff --git a/clients/instrumental-music-center/README.md b/clients/instrumental-music-center/README.md new file mode 100644 index 0000000..372a832 --- /dev/null +++ b/clients/instrumental-music-center/README.md @@ -0,0 +1,59 @@ +# Instrumental Music Center (IMC) + +Music retail + repair shop running AIMsi point-of-sale on-prem. + +## Infrastructure + +### Primary server: IMC1 (192.168.0.2) +- **OS:** Windows Server 2016 Standard (build 14393.7426) +- **Role:** Domain Controller (IMC.local), file server, AIMsi SQL host, RDS host +- **Hardware:** Dell R720, 4 physical cores +- **Disks:** + - `C:` — OS + IIS + a few apps (419 GB, ~77% full as of 2026-04-13) + - `E:` — SQL backups, app installers, Server 2016 install media (`E:\W2016`) + - `F:` — Windows Image Backups + - `S:` — Dedicated SSD (Samsung 850 PRO 256 GB), now holding AIMsi SQL DBs + +### Access +- **SSH:** `ssh IMC\guru@192.168.0.2` (ed25519 key auth; PowerShell default shell) +- **VPN:** OpenVPN `.ovpn` profile (subnet issues with Tailscale 192.168.0.0/24 overlap — disconnect Tailscale first) +- **Domain admin:** `IMC\guru` +- **AIMSQL sysadmin:** `IMC\guru` (added 2026-04-12 via single-user recovery) + +### AIMsi / SQL +- **Instance:** `IMC1\AIMSQL` (MSSQL15 = SQL Server 2019 Express, despite folder name) +- **Databases on `S:\SQL\Data\`:** + - `AIM.mdf` (~8 GB) — production AIMsi database + - `IMC.mdf` (~9 GB) — legacy, usage unclear (kept out of caution) + - `TestConv61223.mdf` (~8 GB) — leftover from 2023-06-12 migration test; safe to drop + - `tempdb.mdf` +- **System DBs remain on** `C:\Program Files\Microsoft SQL Server\MSSQL15.AIMSQL\MSSQL\DATA\` (master, model, msdb) + +### Backups +- **Local SQL backups:** `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\IMCAIM_*.bak` (nightly at 22:00) +- **Retention:** Automated via `C:\Scripts\Clean-AimsiBackups.ps1` scheduled task `IMC AIMsi Backup Retention` (daily 23:30, runs as SYSTEM) +- **Policy:** Last 14 dailies + 1st-of-month; safety override keeps 3 newest regardless +- **Off-site:** Cloudberry/MSP360 "Online Backup" at `C:\ProgramData\Online Backup\` + +### AIM client share +- `\\IMC1\AIM` → `S:\AIM` (4 connected users typical) +- AIM.exe is a 128 KB launcher; real work happens against `IMC1\AIMSQL` +- `RequireSecuritySignature = True` in SMB server config — adds auth overhead + +### Known issues +- **Component store corrupted** (0x80073701 during RDS role removal). KB5075999 re-apply succeeds but rolls back on reboot due to ETW manifest error (HRESULT 15010, provider GUID `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}`) +- `RDS removal is blocked` → pending 2019 migration strategy (in-place vs. clean) +- Oversized `COMPONENTS` hive (~168 MB, normal is 30-50 MB) +- `SMB1 enabled` on server — should disable as security hygiene + +### Other servers in AD +- `IMC2` — 2016 Essentials, last logon 2023, likely decommissioned +- `IMC-VM` — 2016 Standard, last logon 2021, dead +- `SERVERIMC` (192.168.0.63) — SSH-only, 2016 Essentials per AD, state unclear + +## Open work + +- Decide Server 2019 migration path (in-place vs. clean build + migrate) +- Consider dropping `TestConv61223` DB after verifying nothing references it +- Disable SMB1 +- Add IMC vault entry for SSH/SQL/domain credentials diff --git a/clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md b/clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md new file mode 100644 index 0000000..9d2695e --- /dev/null +++ b/clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md @@ -0,0 +1,77 @@ +# Session Log: 2026-04-12 — IMC1 Cleanup, SSH Setup, SQL Move + +## Summary + +Originally engaged to help remove RDS from IMC1 as prep for a Server 2019 upgrade. Removal failed with `0x80073701` (component store corruption). Spent most of the session setting up SSH access, diagnosing the corruption, performing SQL backup cleanup and DB relocation, and ultimately parking the RDS removal as a deeper problem than scoped. + +## Work Completed + +### Remote access +- Installed OpenSSH Server on IMC1 via GitHub release (built-in `Add-WindowsCapability` install was a ghost — binaries never landed due to component store corruption) +- Registered `sshd` and `ssh-agent` services, opened firewall port 22 +- Added public key to `C:\ProgramData\ssh\administrators_authorized_keys` with correct ACLs (inheritance off, Administrators + SYSTEM full control) +- Set PowerShell as default SSH shell via registry +- Diagnosed routing conflict: Tailscale's `pfsense-2` was advertising `192.168.0.0/24` with lower metric than OpenVPN; disconnecting Tailscale restored IMC reachability + +### SQL backup cleanup +- Inventoried `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\`: 66 AIMsi nightly fulls totaling **905 GB** (Feb 1 → Apr 11, 2026) +- Confirmed Cloudberry off-site exists before deletion +- Applied GFS retention manually: kept 14 dailies + 1st-of-month (16 files / 189 GB); deleted 50 files / **716 GB freed on E:** +- Noted size drop from ~15 GB → ~11 GB around 2026-03-28 suggests someone purged/archived data that day + +### Automated retention +- Wrote `C:\Scripts\Clean-AimsiBackups.ps1` implementing GFS policy +- Safety: 3-newest override, filename-pattern guard, log to `C:\Scripts\Logs\aimsi-retention-YYYYMM.log` +- Registered scheduled task `IMC AIMsi Backup Retention`: daily 23:30, SYSTEM, highest privileges, 1h execution limit +- Test ran successfully + +### SQL database relocation (C: → S:) +- Elevated `IMC\guru` to sysadmin on `AIMSQL` instance via single-user recovery mode (net stop → `net start MSSQL$AIMSQL /mSQLCMD` → `ALTER SERVER ROLE sysadmin ADD MEMBER` → normal restart) +- Moved user databases via `ALTER DATABASE ... SET OFFLINE / MODIFY FILE / SET ONLINE`: + - `AIM` (8.6 GB) + - `IMC` (9.8 GB) + - `TestConv61223` (8.8 GB) — still hanging on; candidate for drop +- Moved `tempdb` via `ALTER DATABASE tempdb MODIFY FILE` + service restart; cleaned up orphaned files on C: +- Left system DBs (master, model, msdb) on C: — moving `master` requires startup-parameter changes, marginal benefit +- **Result:** C: 322→278 GB used, S: 27→53 GB used; AIM client launch tested working + +### Minor fix +- Recreated missing `C:\Users\guru\Downloads` folder (registry pointed there, folder didn't exist) + +## RDS Removal / Component Store (parked) + +Root error: `0x80073701 ERROR_SXS_ASSEMBLY_MISSING` on RDS role removal. + +Attempts made: +1. `DISM /Online /Cleanup-Image /RestoreHealth` — failed Error 14 (really `E_OUTOFMEMORY 0x8007000e` from oversized 168 MB COMPONENTS hive) +2. With explicit `/ScratchDir` — failed `E_ACCESSDENIED` (BITS + wuauserv were stopped; DISM couldn't fetch payloads) +3. Started BITS/wuauserv, retried — failed again; BITS idle-auto-stops on Server 2016 (known) +4. `/Source:WIM:E:\W2016\sources\install.wim:2 /LimitAccess` — failed `CBS_E_SOURCE_MISSING` (E:\W2016 is RTM 14393.0 media; damaged assembly is from a post-RTM CU) +5. Extracted KB5075999 (Feb 2026 CU) from local MSU at `C:\Users\guru\Documents\Downloads\` → `DISM /Add-Package` → **staged successfully (S_OK)** but on reboot, apply phase failed with `HRESULT_FROM_WIN32(15010) ERROR_EVT_INVALID_EVENT_DATA` at `onecore\admin\wmi\events\config\manproc.cpp line 733` — ETW event manifest for provider GUID `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}` is malformed → `CBS_E_INSTALLERS_FAILED` → full rollback + +Decision: deeper than scoped. Server otherwise healthy. RDS removal is blocking a planned 2019 upgrade. + +## Next actions (for next session) + +- **Decide 2019 upgrade strategy:** + - Path A: identify specific KB owning provider GUID `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}`, re-register its manifest via `wevtutil im`, retry CU apply + - Path B: try in-place Server 2019 upgrade despite corruption — OS files get rewritten wholesale + - Path C: clean 2019 build + AD/SQL/file/RDS migration +- Verify whether `IMC` database (9.8 GB) is actively used; drop if not +- Verify `TestConv61223` can be dropped safely (leftover migration test from 2023-06-12) +- Disable SMB1 (security hygiene): `Set-SmbServerConfiguration -EnableSMB1Protocol $false` +- Add IMC entry to SOPS vault + +## Key Files and Paths + +- SSH key authorized: `C:\ProgramData\ssh\administrators_authorized_keys` (ed25519 `guru@DESKTOP-0O8A1RL`) +- Retention script: `C:\Scripts\Clean-AimsiBackups.ps1` +- Retention logs: `C:\Scripts\Logs\aimsi-retention-YYYYMM.log` +- DISM scratch: `C:\DISMScratch` +- Expanded KB5075999 payload: `C:\DISMScratch\KB5075999\` +- Local Server 2016 media: `E:\W2016\sources\install.wim` (RTM 14393.0, index 2 = Standard Desktop Experience) + +## Credentials Referenced + +- `IMC\guru` — domain admin, AIMSQL sysadmin. Password handled verbally, not stored here. +- `sa` on `AIMSQL` — exists, enabled, password unknown (tried one candidate, failed — no lockout policy was hit) diff --git a/clients/valleywide/README.md b/clients/valleywide/README.md new file mode 100644 index 0000000..f49398f --- /dev/null +++ b/clients/valleywide/README.md @@ -0,0 +1,52 @@ +# Valleywide (VWP) + +## Infrastructure + +### Servers + +**VWP_ADSRVR (192.168.0.25)** +- Windows Server 2019 Standard (build 17763) +- Domain Controller for `vwp.local` +- SSH enabled (OpenSSH Server), key auth working for `vwp\guru` + +**VWP-QBS (172.16.9.169)** +- Windows Server 2022 Standard +- Internal network only (172.16.9.0/24 reachable via VWP site VPN) +- Runs QuickBooks + **IIS with RD Gateway / RD Web Access** (`/RDWeb`, `/RDWeb/Pages`, `/RDWeb/Feed`, `/Rpc`, `/RpcWithCert`) +- WinRM available on 5985 (used for remote admin via Invoke-Command) + +### Networks +- Internal: `172.16.9.0/24` +- One subnet also numbered `192.168.0.0/24` (conflicts with IMC's LAN if VPNs overlap — be careful switching contexts) + +### Access +- **SSH to VWP_ADSRVR:** `ssh vwp\guru@192.168.0.25` (ed25519 key, added 2026-04-13) +- **Double-hop to VWP-QBS:** SSH won't forward Kerberos; use `Invoke-Command -ComputerName VWP-QBS -Credential $cred` with `vwp\sysadmin` PSCredential + +## Security posture + +### 2026-04-13 incident +RDWeb (`https://VWP-QBS/RDWeb/Pages/login.aspx`) was exposed to the public internet via UDM port forward. Distributed brute-force attack was in progress (multiple external IPs, ~6 POSTs/min, hitting usernames like `scanner`, `Guest`, etc.). This was discovered while investigating repeated `scanner` account lockouts (event 4740) which originally looked like a stale service credential. + +**Actions taken:** +- UDM port forward removed (user action) +- IIS reset on VWP-QBS to drain in-flight attacker sessions +- Domain lockout policy restored (threshold 5, 16-min duration/window) after being temporarily disabled during diagnosis +- 30-day audit: **no successful external logons** — no compromise + +### Current state +- RDWeb no longer reachable from public internet +- Internal access still works on port 443 from within 172.16.9.0/24 +- Account lockout policy active + +### Recommendations (outstanding) +- If RDWeb must be public again: deploy **IPBan** (https://github.com/DigitalRuby/IPBan) + firewall restriction to known client IPs +- Audit UDM for UPnP (prevents the server from re-punching its own hole) +- Consider 2FA / Conditional Access on any externally-reachable Windows service +- Rotate `scanner` AD account password (last set 2024-10-17) as hygiene + +## Open items + +- Confirm UPnP state on UDM +- Document intended RDWeb access pattern (who connects from where) +- Add Valleywide entry to SOPS vault diff --git a/clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md b/clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md new file mode 100644 index 0000000..0e40b0a --- /dev/null +++ b/clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md @@ -0,0 +1,59 @@ +# Session Log: 2026-04-13 — RDWeb Brute-Force Incident + +## Summary + +Originally asked to help find a Windows Server 2016 box that could serve as a DISM source for IMC's broken component store. Valleywide's `VWP_ADSRVR` turned out to be Server 2019 (wrong version), so not useful for IMC — but while investigating, we uncovered an active brute-force attack on Valleywide's publicly-exposed RDWeb and pivoted to incident response. + +No compromise identified. Attack surface closed. + +## Timeline + +1. **Asked user for SSH access** — user provided the `sysadmin` local password and instructions to enable SSH on `VWP_ADSRVR` +2. Added public key to `C:\ProgramData\ssh\administrators_authorized_keys`; key auth landed as `vwp\guru` (domain admin) +3. Discovered server is **Server 2019**, not 2016 — unusable as DISM source for IMC +4. User pivoted: "a number of accounts are/were locked out" +5. Queried AD: lockout policy 5/16min/16min; **`scanner` being locked out every ~20 min, 24/7** from VWP-QBS; also `Receptionist` once and `Guest` twice +6. Initially hypothesized stale scanner credential on some device; checked VWP-QBS via `Invoke-Command` with `vwp\sysadmin`: + - No services or scheduled tasks running as `scanner` + - No stored credentials (`cmdkey /list` empty) + - **4625 failed logons showed `w3wp.exe` as the caller process** (IIS worker) +7. Examined IIS config — no app pool running as scanner, no config file referenced scanner +8. Checked IIS access logs (`C:\inetpub\logs\LogFiles\W3SVC1\u_ex260413.log`) — **found distributed attack in progress**: `POST /RDWeb/Pages/en-US/login.aspx` from dozens of public IPs (China, Belarus, UAE, etc.) at ~6 req/min +9. User removed the UDM port forward exposing 443 to the internet +10. Attack traffic kept arriving briefly (in-flight connections); performed `iisreset` on VWP-QBS to drain +11. Verified: no IIS log activity after 17:15:28, no external established connections on 443 +12. Re-enabled domain lockout policy (had temporarily disabled at user's request during diagnosis) +13. Ran 30-day 4624 audit for public IPv4 source addresses — **zero successful external logons** + +## Key finding + +The `scanner` and `Guest` lockouts had nothing to do with internal stale credentials. They were the brute-force attacker trying common Windows usernames through the public-facing RDWeb portal. Lockout threshold 5 meant every 5 external attempts at `scanner` would trip the lockout, account auto-unlocked after 16 min, repeat. + +Attacker source IPs observed (partial list, all public): +`175.27.166.65`, `1.13.91.38`, `124.220.25.11`, `116.63.167.144`, `213.184.204.221`, `49.235.60.135`, `150.158.14.111`, `129.211.14.197`, `123.207.6.38`, `111.231.15.117`, `217.164.235.215`, `203.143.83.36`, `42.193.102.227`, `124.71.205.87`, `81.70.13.85`, `139.9.90.166`, `42.192.195.29`, `177.21.61.100`, `146.135.5.89` + +(Mix looks consistent with commodity botnet / residential proxy infrastructure — typical of opportunistic RDWeb sweeps.) + +## Actions taken + +- SSH key auth added for `guru@VWP_ADSRVR` +- Default SSH shell: still cmd.exe (not changed — remote work used `powershell -NoProfile -Command` wrappers) +- Domain lockout policy: **temporarily set threshold=0** (disabled) during diagnosis → **restored to 5 / 16min / 16min** once attack cause was understood and UDM change was in place +- IIS reset on VWP-QBS to drain attacker sessions (inetsrv W3SVC/WAS restarted) + +## Decisions / rationale + +- **Disabling lockout was a mistake in retrospect** — I did it assuming stale-credential loop before seeing the attack. Once external source was identified, restored immediately. Window: ~15 minutes. +- **Did not install IPBan** — user chose to close exposure at the edge (UDM) instead. Appropriate since no documented need for public RDWeb was confirmed. IPBan recommended as a prerequisite if RDWeb is ever re-exposed. + +## Outstanding + +- Audit UDM for UPnP (could let the server re-punch a hole) +- Document who actually needs RDWeb access and from where; if external is needed, require VPN + IPBan +- Rotate `scanner` account password as hygiene (PasswordLastSet 2024-10-17) +- Investigate the `LastLogonDate: 9/28/2049` ghost on VWP-QBS AD object — likely time-skew artifact, cosmetic + +## Credentials referenced + +- `vwp\sysadmin` — used for `Invoke-Command` double-hop from VWP_ADSRVR to VWP-QBS. Password handled verbally, not stored here. +- `vwp\guru` — domain admin, SSH key auth. diff --git a/session-logs/2026-04-13-session.md b/session-logs/2026-04-13-session.md new file mode 100644 index 0000000..2347f74 --- /dev/null +++ b/session-logs/2026-04-13-session.md @@ -0,0 +1,152 @@ +# Session Log: 2026-04-13 — Multi-client day + +Long mixed-client session. Work per client is in dedicated logs; this file is the day's index + credential stash. + +## Per-client / per-project logs from today + +- **IMC (Instrumental Music Center)**: `clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md` — main IMC work happened 2026-04-12 but DISM rollback chasing and the client documentation were finished today +- **Valleywide**: `clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md` — security incident +- **Dataforth**: `clients/dataforth/session-logs/2026-04-13-session.md` — API planning + Hoffman call prep + +## One-line per-client summary + +### IMC +- Component store corruption preventing RDS removal and 2019 upgrade +- KB5075999 `/Add-Package` staged successfully but apply-on-boot failed at ETW event manifest for provider `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}` → full rollback +- Parked the RDS removal; server otherwise healthy +- Cleaned up 716 GB of old SQL backups on E: +- Wrote `C:\Scripts\Clean-AimsiBackups.ps1` + scheduled task for GFS retention +- Moved 4 SQL DBs (AIM, IMC, TestConv61223, tempdb) from C: to S: +- Elevated `IMC\guru` to AIMSQL sysadmin via single-user recovery +- Set up SSH access on IMC1 with ed25519 key +- Created `clients/instrumental-music-center/` folder + vault entry `clients/imc/imc1.sops.yaml` + +### Valleywide +- Investigating repeated `scanner` account lockouts turned up an active **brute-force attack on public RDWeb** (`VWP-QBS` at 172.16.9.169) +- User removed UDM port forward; IIS reset to drain in-flight sessions +- 30-day audit: **zero successful external logons — no breach** +- Temporarily disabled domain lockout (mistake in retrospect, was restored within ~15 min) +- Added SSH key to `VWP_ADSRVR` (192.168.0.25); double-hop to VWP-QBS works via `Invoke-Command` + explicit PSCredential +- Created `clients/valleywide/` folder + vault entry `clients/vwp/adsrvr.sops.yaml` (note: sits alongside existing `vwp/dc1.sops.yaml`; IP differs, needs reconciliation next visit) + +### Dataforth +- Reviewed Swagger spec for the new datasheet API +- Confirmed OAuth2 auth_code+PKCE flow (will request `client_credentials` grant for our uploader) +- Prepared question list for John Hoffman Zoom call (batch size, rate limits, idempotency, cutover plan, PDF handling, structured-record vs raw-file push) +- Hoffman will send OAuth credentials today +- No code changes yet — waiting on creds + +### Miscellaneous +- Helped user triage Neptune Exchange (tsorensen → external bounce) — user resolved on their own before I connected +- Explained Defender exclusion commands for git performance (Defender vs git interference) + +## Credentials used today + +> Stored here for quick recovery. Full encrypted entries in `D:\vault\` (age/SOPS). + +### IMC +- **IMC1** (192.168.0.2) domain admin: `IMC\guru` / `r3tr0gradE99!` +- SSH auth: ed25519 key (`guru@DESKTOP-0O8A1RL`) in `C:\ProgramData\ssh\administrators_authorized_keys` +- `AIMSQL` sysadmin: `IMC\guru` (added 2026-04-12 via single-user recovery) +- Vault entry: `D:\vault\clients\imc\imc1.sops.yaml` + +### Valleywide +- **VWP_ADSRVR** (192.168.0.25) SSH: `vwp\guru` (key auth) +- **VWP_ADSRVR / VWP-QBS** domain admin: `vwp\sysadmin` / `r3tr0gradE99#` +- SSH key in `C:\ProgramData\ssh\administrators_authorized_keys` on `VWP_ADSRVR` +- Vault entries (existing, not modified): `vwp/dc1`, `vwp/quickbooks-server-idrac`, `vwp/udm`, `vwp/xenserver` +- Vault entry (added today): `D:\vault\clients\vwp\adsrvr.sops.yaml` + +### Neptune (Dataforth Exchange) +- `neptune.acghosting.com` (67.206.163.124): `ACG\administrator` / `Gptf*77ttb##` +- Access: WinRM NTLM over VPN; requires TrustedHosts on client side +- Vault: `D:\vault\clients\dataforth\neptune-exchange.sops.yaml` (existing) + +### Dataforth API +- OAuth creds pending from Hoffman (expected 2026-04-13) +- Swagger's own client (not for our use): `client_id = dataforth.swagger` +- Old upload path (being retired): `DataforthWebShare` / `Data6277` + +## Key commands / techniques captured + +### Remote shell quirks +- `$` chars in Windows service names (e.g. `MSSQL$AIMSQL`) get eaten by bash when tunneled through SSH → PowerShell. Escape as `\$AIMSQL` in the bash-level string. +- Backticks in PowerShell here-strings can break the bash outer layer. Write to a file with `Write` and run with `powershell -File` for anything non-trivial. +- When SSH-ing into Windows OpenSSH and dispatching to a SECOND host via `Invoke-Command`, key auth doesn't carry Kerberos → need explicit PSCredential. Example: + + $pw = ConvertTo-SecureString 'r3tr0gradE99#' -AsPlainText -Force + $cred = New-Object System.Management.Automation.PSCredential('vwp\sysadmin', $pw) + Invoke-Command -ComputerName VWP-QBS -Credential $cred -ScriptBlock { ... } + +### SQL Server single-user recovery to grant sysadmin +When Windows admin isn't already a sysadmin on an instance: + + Stop-Service 'MSSQL$AIMSQL' -Force + Stop-Service 'MSSQLFDLauncher$AIMSQL' -Force -ErrorAction SilentlyContinue + net start 'MSSQL$AIMSQL' /mSQLCMD + # Connect as any local admin (granted sysadmin in -m mode): + sqlcmd -S localhost\AIMSQL -E -Q "CREATE LOGIN [DOMAIN\user] FROM WINDOWS; ALTER SERVER ROLE sysadmin ADD MEMBER [DOMAIN\user];" + Stop-Service 'MSSQL$AIMSQL' -Force + Start-Service 'MSSQL$AIMSQL' + Start-Service 'MSSQLFDLauncher$AIMSQL' + +### Move SQL database files +Per user database: + + ALTER DATABASE [dbname] SET OFFLINE WITH ROLLBACK IMMEDIATE; + ALTER DATABASE [dbname] MODIFY FILE (NAME=, FILENAME='new\path\file.mdf'); + -- physically move the file on disk + ALTER DATABASE [dbname] SET ONLINE; + +tempdb is different: `MODIFY FILE` + service restart; service recreates files at new location automatically. Delete old tempdb files from original path. + +### Windows OpenSSH key auth for admin accounts +Admin-group users share one key file: + + $authFile = 'C:\ProgramData\ssh\administrators_authorized_keys' + Set-Content -Path $authFile -Value 'ssh-ed25519 AAAA... user@host' -Encoding ASCII + icacls $authFile /inheritance:r + icacls $authFile /grant "Administrators:F" "SYSTEM:F" + Restart-Service sshd + +### DISM repair from a KB cab (when WU broken/blocked) +Expand MSU, then DISM /Add-Package: + + expand -f:* windows10.0-kb5075999-x64_...msu C:\DISMScratch\KB5075999 + DISM /Online /Add-Package /PackagePath:C:\DISMScratch\KB5075999\Windows10.0-KB5075999-x64.cab /ScratchDir:C:\DISMScratch + +## Open / pending items + +### IMC +- Decide 2019 migration path: in-place vs. clean +- Consider dropping `TestConv61223` DB (leftover from 2023-06-12 test) +- Verify `IMC` DB (9.8 GB) usage; drop if dead +- Disable SMB1 (`Set-SmbServerConfiguration -EnableSMB1Protocol $false`) + +### Valleywide +- Audit UDM for UPnP (prevents the server from re-punching a hole) +- Rotate `scanner` AD account password (last set 2024-10-17) +- Investigate `LastLogonDate: 9/28/2049` ghost on VWP-QBS AD object (cosmetic) +- If RDWeb needs to go public again: IPBan + IP allowlist first +- Reconcile `vwp/adsrvr.sops.yaml` (new) vs `vwp/dc1.sops.yaml` (existing) — may be same server multi-homed, or separate DC + +### Dataforth +- Await OAuth creds from Hoffman +- Store creds in `D:\vault\clients\dataforth\dataforth-api-oauth.sops.yaml` when received +- Push back for `client_credentials` grant on a dedicated uploader client +- Build POC uploader (get token → POST one file → GET + verify) +- Plan initial backfill of 501K files + +## Vault changes + +- Created: `D:\vault\clients\imc\imc1.sops.yaml` (encrypted) +- Created: `D:\vault\clients\vwp\adsrvr.sops.yaml` (encrypted) + +## Documentation changes + +- Created: `clients/instrumental-music-center/README.md` +- Created: `clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md` +- Created: `clients/valleywide/README.md` +- Created: `clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md` +- Created: `clients/dataforth/session-logs/2026-04-13-session.md` +- Created: this file