azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Session log: IMC SQL move + DISM repair attempt, VWP RDWeb brute-force incident, Dataforth API planning

Browse Source 
- IMC: document 716 GB SQL backup cleanup, retention scheduled task, DB move C:->S:, sysadmin grant via single-user recovery, parked RDS removal after KB5075999 apply rolled back on ETW manifest error
- Valleywide: document RDWeb brute-force incident on VWP-QBS, UDM port forward closure, 30-day audit showing no breach, lockout policy restoration
- Dataforth: capture Swagger API review and Hoffman Zoom call prep
This commit is contained in:
Mike Swanson
2026-04-13 15:40:43 -07:00
parent a78fb96f95
commit 5169936cfc
6 changed files with 518 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

119
clients/dataforth/session-logs/2026-04-13-session.md Normal file
View File

@@ -0,0 +1,119 @@
# Session Log: 2026-04-13 — Dataforth
## Summary
Continuation of the test datasheet pipeline work. Prior session (2026-04-12) confirmed PostgreSQL migration complete; Hoffman provided the new Swagger API URL; awaiting OAuth credentials. Today: reviewed the full API spec, prepared a structured question list for a Zoom call with Hoffman, and discussed architecture options (raw file upload vs. structured record push vs. direct DB).
Also helped user triage an unrelated Neptune Exchange mail-flow issue (tsorensen → external bounce). User resolved on their own before I got into it.
## Work completed
### API spec review
Pulled `https://www.dataforth.com/swagger/v1/swagger.json` and mapped endpoints.
**Base URL:** `https://www.dataforth.com` (presumed; Swagger UI at `/swagger/index.html`)
**Authentication (IdentityServer-style)**
- Flow: **OAuth2 Authorization Code + PKCE**
- Authorization URL: `https://login.dataforth.com/connect/authorize`
- Token URL: `https://login.dataforth.com/connect/token`
- Scopes: `openid`, `profile`, `dataforth.web`
- Swagger's own test client: `client_id = dataforth.swagger` (NOT for our use)
- OIDC discovery expected at: `https://login.dataforth.com/.well-known/openid-configuration`
**All endpoints**
| Path | Method |
|------|--------|
| `/api/v1/Admin/refresh-cache` | POST |
| `/api/v1/Admin/cache-status` | GET |
| `/api/v1/Categories` | GET |
| `/api/v1/Categories/{id}` | GET |
| `/api/v1/Categories/by-catalog-node/{catalogNodeId}` | GET |
| `/api/v1/OrderableProducts/{orderableProductId}/Attributes` | POST |
| `/api/v1/OrderableProducts/{orderableProductId}/Attributes/{attributeId}` | PUT/DELETE |
| `/api/v1/Products`, `/{id}`, `/by-part-number/{partNumber}` | GET |
| `/api/v1/product-series`, `/{id}`, `/by-designation/{designation}`, `/by-catalog-node/{catalogNodeId}` | GET |
| `/api/v1/ProductType`, `/{productTypeId}/products` | GET |
| `/api/v1/TestReportDataFiles` | POST (single upload) |
| `/api/v1/TestReportDataFiles` | GET (paginated list) |
| `/api/v1/TestReportDataFiles/bulk` | POST (batch upload) |
| `/api/v1/TestReportDataFiles/{serialNumber}` | GET / DELETE |
| `/api/v1/TestReportDataFiles/stats` | GET |
**TestReportDataFiles payload shapes**
- POST single: `{ SerialNumber: string(max 50), Content: string(min 1) }` → `{ SerialNumber, ContentHash, Created }`
- POST bulk: `{ Items: [CreateTestReportRequest, ...] }` → `{ TotalReceived, Created, Updated, Unchanged, Errors[] }`
- GET single: `{ SerialNumber, Content, CreatedAtUtc, UpdatedAtUtc }`
- GET stats: `{ TotalCount, LatestCreatedAtUtc, LatestUpdatedAtUtc }`
- Server handles dedup via ContentHash → client doesn't need to pre-check.
### Architecture discussion
Three options for delivering datasheets:
- **A: Raw file blob via current API** — works today, zero new API work, simple client code
- **B: Structured records via new endpoints** — cleaner long-term; we already have parsed data in AD2's PostgreSQL `TestDataDB` (2.8M records post-2026-04-12 migration). Requires Hoffman to add endpoints
- **C: Direct DB access** — rejected (coupling, security, DBA nightmare)
Preferred path: whichever is less work for Hoffman. Frame it as offering flexibility — we can send raw text, structured JSON, or even CSV.
### Questions prepared for John Hoffman Zoom call
Produced a prioritized list (MUST / SHOULD / NICE) covering:
- Batch size + payload size + rate limits (MUST)
- Idempotency + dedup semantics (MUST)
- Cutover plan from old DataforthWebShare path (MUST)
- Request: enable `client_credentials` grant on a new client for the AD2 uploader (SHOULD)
- Staging endpoint availability (SHOULD)
- PDF handling (`X:\For_Web_PDF`) — same endpoint or different? (SHOULD)
- Product linkage — does a TestReport need to link to a Product/Series record? (SHOULD)
- Monitoring + error visibility on his side (NICE)
- SLA / escalation contact (NICE)
### Pending from Hoffman (as of end-of-session 2026-04-13)
- OAuth credentials (he said "today")
- Clarification on client_credentials grant support
- Answers to the MUST questions above after the Zoom
## Pipeline context (unchanged from 2026-04-12)
### Current state
- **Stage 1**: DOS test stations → D2TESTNAS (192.168.0.9, rsync daemon, module "test" → /data/test) ✓
- **Stage 2**: NAS → AD2 via `Sync-FromNAS-rsync.ps1` scheduled every 15 min ✓
- **Stage 3**: DFWDS.exe validates + renames — **config wiped in crypto attack**; `C:\DFWDS\DFWDS_NAMES.TXT` missing. Check Haubner D: for backup.
- **Stage 4**: Website upload — **BROKEN**; this is what we're rebuilding via the new API
- **Stage 5**: PDF generation — ~4,773 PDFs in `X:\For_Web_PDF`, origin unclear
### Data locations
- Incoming: `X:\Test_Datasheets` (staging)
- Validated: `X:\For_Web` (~501K files) ← uploader source
- PDFs: `X:\For_Web_PDF` (~4.7K files)
- Rejected: `X:\Bad_Datasheets` (~18K)
- DFWDS logs: `X:\Datasheets_Log`
- `X:` = `\\ad2\webshare`
### Datasheet format
Plain text, ~50 lines. Header: Dataforth address/phone. Fields: Date, Model (e.g. SCM5B41-03), SN (e.g. 178439-1), accuracy test table, final test results. Filename: `{SN}.txt` (e.g. `178439-1.txt`).
### Credentials used/referenced
- **Old upload path** (being replaced): `DataforthWebShare / Data6277`
- **New API**: OAuth client credentials pending from Hoffman
- **Neptune Exchange** (for today's mail triage): `ACG\administrator` / `Gptf*77ttb##` — requires VPN
## Next session plan
1. Receive OAuth creds from Hoffman (client_id + client_secret, ideally client_credentials grant enabled)
2. Store credentials in `D:\vault\clients\dataforth\dataforth-api-oauth.sops.yaml`
3. Stand up a one-page POC: get token, POST one test report, verify via GET
4. If POC works → implement full uploader on AD2:
- Language: PowerShell (fits existing scripts) or Python (already used in `projects/dataforth-dos/datasheet-pipeline/implementation/`)
- State tracking: local manifest (serial → hash + last-upload-time) or use server's ContentHash response
- Use `/bulk` endpoint in batches (size TBD with Hoffman)
- Scheduled task on AD2, 15-min or hourly cadence
- Initial backfill script for 501K files — run off-hours
5. Parallel-run with old webshare path until confident, then retire old path
## Reference URLs
- Swagger UI: https://www.dataforth.com/swagger/index.html
- Swagger JSON: https://www.dataforth.com/swagger/v1/swagger.json
- Authorization URL: https://login.dataforth.com/connect/authorize
- Token URL: https://login.dataforth.com/connect/token
- Expected OIDC discovery: https://login.dataforth.com/.well-known/openid-configuration

59
clients/instrumental-music-center/README.md Normal file
View File

@@ -0,0 +1,59 @@
# Instrumental Music Center (IMC)
Music retail + repair shop running AIMsi point-of-sale on-prem.
## Infrastructure
### Primary server: IMC1 (192.168.0.2)
- **OS:** Windows Server 2016 Standard (build 14393.7426)
- **Role:** Domain Controller (IMC.local), file server, AIMsi SQL host, RDS host
- **Hardware:** Dell R720, 4 physical cores
- **Disks:**
- `C:` — OS + IIS + a few apps (419 GB, ~77% full as of 2026-04-13)
- `E:` — SQL backups, app installers, Server 2016 install media (`E:\W2016`)
- `F:` — Windows Image Backups
- `S:` — Dedicated SSD (Samsung 850 PRO 256 GB), now holding AIMsi SQL DBs
### Access
- **SSH:** `ssh IMC\guru@192.168.0.2` (ed25519 key auth; PowerShell default shell)
- **VPN:** OpenVPN `.ovpn` profile (subnet issues with Tailscale 192.168.0.0/24 overlap — disconnect Tailscale first)
- **Domain admin:** `IMC\guru`
- **AIMSQL sysadmin:** `IMC\guru` (added 2026-04-12 via single-user recovery)
### AIMsi / SQL
- **Instance:** `IMC1\AIMSQL` (MSSQL15 = SQL Server 2019 Express, despite folder name)
- **Databases on `S:\SQL\Data\`:**
- `AIM.mdf` (~8 GB) — production AIMsi database
- `IMC.mdf` (~9 GB) — legacy, usage unclear (kept out of caution)
- `TestConv61223.mdf` (~8 GB) — leftover from 2023-06-12 migration test; safe to drop
- `tempdb.mdf`
- **System DBs remain on** `C:\Program Files\Microsoft SQL Server\MSSQL15.AIMSQL\MSSQL\DATA\` (master, model, msdb)
### Backups
- **Local SQL backups:** `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\IMCAIM_*.bak` (nightly at 22:00)
- **Retention:** Automated via `C:\Scripts\Clean-AimsiBackups.ps1` scheduled task `IMC AIMsi Backup Retention` (daily 23:30, runs as SYSTEM)
- **Policy:** Last 14 dailies + 1st-of-month; safety override keeps 3 newest regardless
- **Off-site:** Cloudberry/MSP360 "Online Backup" at `C:\ProgramData\Online Backup\`
### AIM client share
- `\\IMC1\AIM` → `S:\AIM` (4 connected users typical)
- AIM.exe is a 128 KB launcher; real work happens against `IMC1\AIMSQL`
- `RequireSecuritySignature = True` in SMB server config — adds auth overhead
### Known issues
- **Component store corrupted** (0x80073701 during RDS role removal). KB5075999 re-apply succeeds but rolls back on reboot due to ETW manifest error (HRESULT 15010, provider GUID `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}`)
- `RDS removal is blocked` → pending 2019 migration strategy (in-place vs. clean)
- Oversized `COMPONENTS` hive (~168 MB, normal is 30-50 MB)
- `SMB1 enabled` on server — should disable as security hygiene
### Other servers in AD
- `IMC2` — 2016 Essentials, last logon 2023, likely decommissioned
- `IMC-VM` — 2016 Standard, last logon 2021, dead
- `SERVERIMC` (192.168.0.63) — SSH-only, 2016 Essentials per AD, state unclear
## Open work
- Decide Server 2019 migration path (in-place vs. clean build + migrate)
- Consider dropping `TestConv61223` DB after verifying nothing references it
- Disable SMB1
- Add IMC vault entry for SSH/SQL/domain credentials

77
clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md Normal file
View File

@@ -0,0 +1,77 @@
# Session Log: 2026-04-12 — IMC1 Cleanup, SSH Setup, SQL Move
## Summary
Originally engaged to help remove RDS from IMC1 as prep for a Server 2019 upgrade. Removal failed with `0x80073701` (component store corruption). Spent most of the session setting up SSH access, diagnosing the corruption, performing SQL backup cleanup and DB relocation, and ultimately parking the RDS removal as a deeper problem than scoped.
## Work Completed
### Remote access
- Installed OpenSSH Server on IMC1 via GitHub release (built-in `Add-WindowsCapability` install was a ghost — binaries never landed due to component store corruption)
- Registered `sshd` and `ssh-agent` services, opened firewall port 22
- Added public key to `C:\ProgramData\ssh\administrators_authorized_keys` with correct ACLs (inheritance off, Administrators + SYSTEM full control)
- Set PowerShell as default SSH shell via registry
- Diagnosed routing conflict: Tailscale's `pfsense-2` was advertising `192.168.0.0/24` with lower metric than OpenVPN; disconnecting Tailscale restored IMC reachability
### SQL backup cleanup
- Inventoried `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\`: 66 AIMsi nightly fulls totaling **905 GB** (Feb 1 → Apr 11, 2026)
- Confirmed Cloudberry off-site exists before deletion
- Applied GFS retention manually: kept 14 dailies + 1st-of-month (16 files / 189 GB); deleted 50 files / **716 GB freed on E:**
- Noted size drop from ~15 GB → ~11 GB around 2026-03-28 suggests someone purged/archived data that day
### Automated retention
- Wrote `C:\Scripts\Clean-AimsiBackups.ps1` implementing GFS policy
- Safety: 3-newest override, filename-pattern guard, log to `C:\Scripts\Logs\aimsi-retention-YYYYMM.log`
- Registered scheduled task `IMC AIMsi Backup Retention`: daily 23:30, SYSTEM, highest privileges, 1h execution limit
- Test ran successfully
### SQL database relocation (C: → S:)
- Elevated `IMC\guru` to sysadmin on `AIMSQL` instance via single-user recovery mode (net stop → `net start MSSQL$AIMSQL /mSQLCMD` → `ALTER SERVER ROLE sysadmin ADD MEMBER` → normal restart)
- Moved user databases via `ALTER DATABASE ... SET OFFLINE / MODIFY FILE / SET ONLINE`:
- `AIM` (8.6 GB)
- `IMC` (9.8 GB)
- `TestConv61223` (8.8 GB) — still hanging on; candidate for drop
- Moved `tempdb` via `ALTER DATABASE tempdb MODIFY FILE` + service restart; cleaned up orphaned files on C:
- Left system DBs (master, model, msdb) on C: — moving `master` requires startup-parameter changes, marginal benefit
- **Result:** C: 322→278 GB used, S: 27→53 GB used; AIM client launch tested working
### Minor fix
- Recreated missing `C:\Users\guru\Downloads` folder (registry pointed there, folder didn't exist)
## RDS Removal / Component Store (parked)
Root error: `0x80073701 ERROR_SXS_ASSEMBLY_MISSING` on RDS role removal.
Attempts made:
1. `DISM /Online /Cleanup-Image /RestoreHealth` — failed Error 14 (really `E_OUTOFMEMORY 0x8007000e` from oversized 168 MB COMPONENTS hive)
2. With explicit `/ScratchDir` — failed `E_ACCESSDENIED` (BITS + wuauserv were stopped; DISM couldn't fetch payloads)
3. Started BITS/wuauserv, retried — failed again; BITS idle-auto-stops on Server 2016 (known)
4. `/Source:WIM:E:\W2016\sources\install.wim:2 /LimitAccess` — failed `CBS_E_SOURCE_MISSING` (E:\W2016 is RTM 14393.0 media; damaged assembly is from a post-RTM CU)
5. Extracted KB5075999 (Feb 2026 CU) from local MSU at `C:\Users\guru\Documents\Downloads\` → `DISM /Add-Package` → **staged successfully (S_OK)** but on reboot, apply phase failed with `HRESULT_FROM_WIN32(15010) ERROR_EVT_INVALID_EVENT_DATA` at `onecore\admin\wmi\events\config\manproc.cpp line 733` — ETW event manifest for provider GUID `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}` is malformed → `CBS_E_INSTALLERS_FAILED` → full rollback
Decision: deeper than scoped. Server otherwise healthy. RDS removal is blocking a planned 2019 upgrade.
## Next actions (for next session)
- **Decide 2019 upgrade strategy:**
- Path A: identify specific KB owning provider GUID `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}`, re-register its manifest via `wevtutil im`, retry CU apply
- Path B: try in-place Server 2019 upgrade despite corruption — OS files get rewritten wholesale
- Path C: clean 2019 build + AD/SQL/file/RDS migration
- Verify whether `IMC` database (9.8 GB) is actively used; drop if not
- Verify `TestConv61223` can be dropped safely (leftover migration test from 2023-06-12)
- Disable SMB1 (security hygiene): `Set-SmbServerConfiguration -EnableSMB1Protocol $false`
- Add IMC entry to SOPS vault
## Key Files and Paths
- SSH key authorized: `C:\ProgramData\ssh\administrators_authorized_keys` (ed25519 `guru@DESKTOP-0O8A1RL`)
- Retention script: `C:\Scripts\Clean-AimsiBackups.ps1`
- Retention logs: `C:\Scripts\Logs\aimsi-retention-YYYYMM.log`
- DISM scratch: `C:\DISMScratch`
- Expanded KB5075999 payload: `C:\DISMScratch\KB5075999\`
- Local Server 2016 media: `E:\W2016\sources\install.wim` (RTM 14393.0, index 2 = Standard Desktop Experience)
## Credentials Referenced
- `IMC\guru` — domain admin, AIMSQL sysadmin. Password handled verbally, not stored here.
- `sa` on `AIMSQL` — exists, enabled, password unknown (tried one candidate, failed — no lockout policy was hit)

52
clients/valleywide/README.md Normal file
View File

@@ -0,0 +1,52 @@
# Valleywide (VWP)
## Infrastructure
### Servers
**VWP_ADSRVR (192.168.0.25)**
- Windows Server 2019 Standard (build 17763)
- Domain Controller for `vwp.local`
- SSH enabled (OpenSSH Server), key auth working for `vwp\guru`
**VWP-QBS (172.16.9.169)**
- Windows Server 2022 Standard
- Internal network only (172.16.9.0/24 reachable via VWP site VPN)
- Runs QuickBooks + **IIS with RD Gateway / RD Web Access** (`/RDWeb`, `/RDWeb/Pages`, `/RDWeb/Feed`, `/Rpc`, `/RpcWithCert`)
- WinRM available on 5985 (used for remote admin via Invoke-Command)
### Networks
- Internal: `172.16.9.0/24`
- One subnet also numbered `192.168.0.0/24` (conflicts with IMC's LAN if VPNs overlap — be careful switching contexts)
### Access
- **SSH to VWP_ADSRVR:** `ssh vwp\guru@192.168.0.25` (ed25519 key, added 2026-04-13)
- **Double-hop to VWP-QBS:** SSH won't forward Kerberos; use `Invoke-Command -ComputerName VWP-QBS -Credential $cred` with `vwp\sysadmin` PSCredential
## Security posture
### 2026-04-13 incident
RDWeb (`https://VWP-QBS/RDWeb/Pages/login.aspx`) was exposed to the public internet via UDM port forward. Distributed brute-force attack was in progress (multiple external IPs, ~6 POSTs/min, hitting usernames like `scanner`, `Guest`, etc.). This was discovered while investigating repeated `scanner` account lockouts (event 4740) which originally looked like a stale service credential.
**Actions taken:**
- UDM port forward removed (user action)
- IIS reset on VWP-QBS to drain in-flight attacker sessions
- Domain lockout policy restored (threshold 5, 16-min duration/window) after being temporarily disabled during diagnosis
- 30-day audit: **no successful external logons** — no compromise
### Current state
- RDWeb no longer reachable from public internet
- Internal access still works on port 443 from within 172.16.9.0/24
- Account lockout policy active
### Recommendations (outstanding)
- If RDWeb must be public again: deploy **IPBan** (https://github.com/DigitalRuby/IPBan) + firewall restriction to known client IPs
- Audit UDM for UPnP (prevents the server from re-punching its own hole)
- Consider 2FA / Conditional Access on any externally-reachable Windows service
- Rotate `scanner` AD account password (last set 2024-10-17) as hygiene
## Open items
- Confirm UPnP state on UDM
- Document intended RDWeb access pattern (who connects from where)
- Add Valleywide entry to SOPS vault

59
clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md Normal file
View File

@@ -0,0 +1,59 @@
# Session Log: 2026-04-13 — RDWeb Brute-Force Incident
## Summary
Originally asked to help find a Windows Server 2016 box that could serve as a DISM source for IMC's broken component store. Valleywide's `VWP_ADSRVR` turned out to be Server 2019 (wrong version), so not useful for IMC — but while investigating, we uncovered an active brute-force attack on Valleywide's publicly-exposed RDWeb and pivoted to incident response.
No compromise identified. Attack surface closed.
## Timeline
1. **Asked user for SSH access** — user provided the `sysadmin` local password and instructions to enable SSH on `VWP_ADSRVR`
2. Added public key to `C:\ProgramData\ssh\administrators_authorized_keys`; key auth landed as `vwp\guru` (domain admin)
3. Discovered server is **Server 2019**, not 2016 — unusable as DISM source for IMC
4. User pivoted: "a number of accounts are/were locked out"
5. Queried AD: lockout policy 5/16min/16min; **`scanner` being locked out every ~20 min, 24/7** from VWP-QBS; also `Receptionist` once and `Guest` twice
6. Initially hypothesized stale scanner credential on some device; checked VWP-QBS via `Invoke-Command` with `vwp\sysadmin`:
- No services or scheduled tasks running as `scanner`
- No stored credentials (`cmdkey /list` empty)
- **4625 failed logons showed `w3wp.exe` as the caller process** (IIS worker)
7. Examined IIS config — no app pool running as scanner, no config file referenced scanner
8. Checked IIS access logs (`C:\inetpub\logs\LogFiles\W3SVC1\u_ex260413.log`) — **found distributed attack in progress**: `POST /RDWeb/Pages/en-US/login.aspx` from dozens of public IPs (China, Belarus, UAE, etc.) at ~6 req/min
9. User removed the UDM port forward exposing 443 to the internet
10. Attack traffic kept arriving briefly (in-flight connections); performed `iisreset` on VWP-QBS to drain
11. Verified: no IIS log activity after 17:15:28, no external established connections on 443
12. Re-enabled domain lockout policy (had temporarily disabled at user's request during diagnosis)
13. Ran 30-day 4624 audit for public IPv4 source addresses — **zero successful external logons**
## Key finding
The `scanner` and `Guest` lockouts had nothing to do with internal stale credentials. They were the brute-force attacker trying common Windows usernames through the public-facing RDWeb portal. Lockout threshold 5 meant every 5 external attempts at `scanner` would trip the lockout, account auto-unlocked after 16 min, repeat.
Attacker source IPs observed (partial list, all public):
`175.27.166.65`, `1.13.91.38`, `124.220.25.11`, `116.63.167.144`, `213.184.204.221`, `49.235.60.135`, `150.158.14.111`, `129.211.14.197`, `123.207.6.38`, `111.231.15.117`, `217.164.235.215`, `203.143.83.36`, `42.193.102.227`, `124.71.205.87`, `81.70.13.85`, `139.9.90.166`, `42.192.195.29`, `177.21.61.100`, `146.135.5.89`
(Mix looks consistent with commodity botnet / residential proxy infrastructure — typical of opportunistic RDWeb sweeps.)
## Actions taken
- SSH key auth added for `guru@VWP_ADSRVR`
- Default SSH shell: still cmd.exe (not changed — remote work used `powershell -NoProfile -Command` wrappers)
- Domain lockout policy: **temporarily set threshold=0** (disabled) during diagnosis → **restored to 5 / 16min / 16min** once attack cause was understood and UDM change was in place
- IIS reset on VWP-QBS to drain attacker sessions (inetsrv W3SVC/WAS restarted)
## Decisions / rationale
- **Disabling lockout was a mistake in retrospect** — I did it assuming stale-credential loop before seeing the attack. Once external source was identified, restored immediately. Window: ~15 minutes.
- **Did not install IPBan** — user chose to close exposure at the edge (UDM) instead. Appropriate since no documented need for public RDWeb was confirmed. IPBan recommended as a prerequisite if RDWeb is ever re-exposed.
## Outstanding
- Audit UDM for UPnP (could let the server re-punch a hole)
- Document who actually needs RDWeb access and from where; if external is needed, require VPN + IPBan
- Rotate `scanner` account password as hygiene (PasswordLastSet 2024-10-17)
- Investigate the `LastLogonDate: 9/28/2049` ghost on VWP-QBS AD object — likely time-skew artifact, cosmetic
## Credentials referenced
- `vwp\sysadmin` — used for `Invoke-Command` double-hop from VWP_ADSRVR to VWP-QBS. Password handled verbally, not stored here.
- `vwp\guru` — domain admin, SSH key auth.

152
session-logs/2026-04-13-session.md Normal file
View File

@@ -0,0 +1,152 @@
# Session Log: 2026-04-13 — Multi-client day
Long mixed-client session. Work per client is in dedicated logs; this file is the day's index + credential stash.
## Per-client / per-project logs from today
- **IMC (Instrumental Music Center)**: `clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md` — main IMC work happened 2026-04-12 but DISM rollback chasing and the client documentation were finished today
- **Valleywide**: `clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md` — security incident
- **Dataforth**: `clients/dataforth/session-logs/2026-04-13-session.md` — API planning + Hoffman call prep
## One-line per-client summary
### IMC
- Component store corruption preventing RDS removal and 2019 upgrade
- KB5075999 `/Add-Package` staged successfully but apply-on-boot failed at ETW event manifest for provider `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}` → full rollback
- Parked the RDS removal; server otherwise healthy
- Cleaned up 716 GB of old SQL backups on E:
- Wrote `C:\Scripts\Clean-AimsiBackups.ps1` + scheduled task for GFS retention
- Moved 4 SQL DBs (AIM, IMC, TestConv61223, tempdb) from C: to S:
- Elevated `IMC\guru` to AIMSQL sysadmin via single-user recovery
- Set up SSH access on IMC1 with ed25519 key
- Created `clients/instrumental-music-center/` folder + vault entry `clients/imc/imc1.sops.yaml`
### Valleywide
- Investigating repeated `scanner` account lockouts turned up an active **brute-force attack on public RDWeb** (`VWP-QBS` at 172.16.9.169)
- User removed UDM port forward; IIS reset to drain in-flight sessions
- 30-day audit: **zero successful external logons — no breach**
- Temporarily disabled domain lockout (mistake in retrospect, was restored within ~15 min)
- Added SSH key to `VWP_ADSRVR` (192.168.0.25); double-hop to VWP-QBS works via `Invoke-Command` + explicit PSCredential
- Created `clients/valleywide/` folder + vault entry `clients/vwp/adsrvr.sops.yaml` (note: sits alongside existing `vwp/dc1.sops.yaml`; IP differs, needs reconciliation next visit)
### Dataforth
- Reviewed Swagger spec for the new datasheet API
- Confirmed OAuth2 auth_code+PKCE flow (will request `client_credentials` grant for our uploader)
- Prepared question list for John Hoffman Zoom call (batch size, rate limits, idempotency, cutover plan, PDF handling, structured-record vs raw-file push)
- Hoffman will send OAuth credentials today
- No code changes yet — waiting on creds
### Miscellaneous
- Helped user triage Neptune Exchange (tsorensen → external bounce) — user resolved on their own before I connected
- Explained Defender exclusion commands for git performance (Defender vs git interference)
## Credentials used today
> Stored here for quick recovery. Full encrypted entries in `D:\vault\` (age/SOPS).
### IMC
- **IMC1** (192.168.0.2) domain admin: `IMC\guru` / `r3tr0gradE99!`
- SSH auth: ed25519 key (`guru@DESKTOP-0O8A1RL`) in `C:\ProgramData\ssh\administrators_authorized_keys`
- `AIMSQL` sysadmin: `IMC\guru` (added 2026-04-12 via single-user recovery)
- Vault entry: `D:\vault\clients\imc\imc1.sops.yaml`
### Valleywide
- **VWP_ADSRVR** (192.168.0.25) SSH: `vwp\guru` (key auth)
- **VWP_ADSRVR / VWP-QBS** domain admin: `vwp\sysadmin` / `r3tr0gradE99#`
- SSH key in `C:\ProgramData\ssh\administrators_authorized_keys` on `VWP_ADSRVR`
- Vault entries (existing, not modified): `vwp/dc1`, `vwp/quickbooks-server-idrac`, `vwp/udm`, `vwp/xenserver`
- Vault entry (added today): `D:\vault\clients\vwp\adsrvr.sops.yaml`
### Neptune (Dataforth Exchange)
- `neptune.acghosting.com` (67.206.163.124): `ACG\administrator` / `Gptf*77ttb##`
- Access: WinRM NTLM over VPN; requires TrustedHosts on client side
- Vault: `D:\vault\clients\dataforth\neptune-exchange.sops.yaml` (existing)
### Dataforth API
- OAuth creds pending from Hoffman (expected 2026-04-13)
- Swagger's own client (not for our use): `client_id = dataforth.swagger`
- Old upload path (being retired): `DataforthWebShare` / `Data6277`
## Key commands / techniques captured
### Remote shell quirks
- `$` chars in Windows service names (e.g. `MSSQL$AIMSQL`) get eaten by bash when tunneled through SSH → PowerShell. Escape as `\$AIMSQL` in the bash-level string.
- Backticks in PowerShell here-strings can break the bash outer layer. Write to a file with `Write` and run with `powershell -File` for anything non-trivial.
- When SSH-ing into Windows OpenSSH and dispatching to a SECOND host via `Invoke-Command`, key auth doesn't carry Kerberos → need explicit PSCredential. Example:
$pw = ConvertTo-SecureString 'r3tr0gradE99#' -AsPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential('vwp\sysadmin', $pw)
Invoke-Command -ComputerName VWP-QBS -Credential $cred -ScriptBlock { ... }
### SQL Server single-user recovery to grant sysadmin
When Windows admin isn't already a sysadmin on an instance:
Stop-Service 'MSSQL$AIMSQL' -Force
Stop-Service 'MSSQLFDLauncher$AIMSQL' -Force -ErrorAction SilentlyContinue
net start 'MSSQL$AIMSQL' /mSQLCMD
# Connect as any local admin (granted sysadmin in -m mode):
sqlcmd -S localhost\AIMSQL -E -Q "CREATE LOGIN [DOMAIN\user] FROM WINDOWS; ALTER SERVER ROLE sysadmin ADD MEMBER [DOMAIN\user];"
Stop-Service 'MSSQL$AIMSQL' -Force
Start-Service 'MSSQL$AIMSQL'
Start-Service 'MSSQLFDLauncher$AIMSQL'
### Move SQL database files
Per user database:
ALTER DATABASE [dbname] SET OFFLINE WITH ROLLBACK IMMEDIATE;
ALTER DATABASE [dbname] MODIFY FILE (NAME=<logical>, FILENAME='new\path\file.mdf');
-- physically move the file on disk
ALTER DATABASE [dbname] SET ONLINE;
tempdb is different: `MODIFY FILE` + service restart; service recreates files at new location automatically. Delete old tempdb files from original path.
### Windows OpenSSH key auth for admin accounts
Admin-group users share one key file:
$authFile = 'C:\ProgramData\ssh\administrators_authorized_keys'
Set-Content -Path $authFile -Value 'ssh-ed25519 AAAA... user@host' -Encoding ASCII
icacls $authFile /inheritance:r
icacls $authFile /grant "Administrators:F" "SYSTEM:F"
Restart-Service sshd
### DISM repair from a KB cab (when WU broken/blocked)
Expand MSU, then DISM /Add-Package:
expand -f:* windows10.0-kb5075999-x64_...msu C:\DISMScratch\KB5075999
DISM /Online /Add-Package /PackagePath:C:\DISMScratch\KB5075999\Windows10.0-KB5075999-x64.cab /ScratchDir:C:\DISMScratch
## Open / pending items
### IMC
- Decide 2019 migration path: in-place vs. clean
- Consider dropping `TestConv61223` DB (leftover from 2023-06-12 test)
- Verify `IMC` DB (9.8 GB) usage; drop if dead
- Disable SMB1 (`Set-SmbServerConfiguration -EnableSMB1Protocol $false`)
### Valleywide
- Audit UDM for UPnP (prevents the server from re-punching a hole)
- Rotate `scanner` AD account password (last set 2024-10-17)
- Investigate `LastLogonDate: 9/28/2049` ghost on VWP-QBS AD object (cosmetic)
- If RDWeb needs to go public again: IPBan + IP allowlist first
- Reconcile `vwp/adsrvr.sops.yaml` (new) vs `vwp/dc1.sops.yaml` (existing) — may be same server multi-homed, or separate DC
### Dataforth
- Await OAuth creds from Hoffman
- Store creds in `D:\vault\clients\dataforth\dataforth-api-oauth.sops.yaml` when received
- Push back for `client_credentials` grant on a dedicated uploader client
- Build POC uploader (get token → POST one file → GET + verify)
- Plan initial backfill of 501K files
## Vault changes
- Created: `D:\vault\clients\imc\imc1.sops.yaml` (encrypted)
- Created: `D:\vault\clients\vwp\adsrvr.sops.yaml` (encrypted)
## Documentation changes
- Created: `clients/instrumental-music-center/README.md`
- Created: `clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md`
- Created: `clients/valleywide/README.md`
- Created: `clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md`
- Created: `clients/dataforth/session-logs/2026-04-13-session.md`
- Created: this file