glaztech: least-privilege tom DB migration scope + 2026-06-05 session log

Scope (v0.3) for replacing the website's sysadmin login 'tom' with a
least-privilege login: two-phase plan (GTIware co-residency forces keeping
cc_file in Phase 1), Grok + Gemini independent review folded in, and live
RMM recon findings that materially changed the picture - the website is a
cross-office + Sage accounting + payroll + msdb hub on one sysadmin
credential, SQL is centralized on GTI-INV-SQL\GTISQL:3436 (not per-site).
PARKED pending a full network recon. Session log covers the website outage
fix (incomplete E1 ACL hardening) + the scoping + recon.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

clients/glaztech/session-logs/2026-06-05-session.md

@@ -0,0 +1,92 @@
+# Glaztech Session Log — 2026-06-05
+
+## User
+- **User:** Mike Swanson (mike)
+- **Machine:** GURU-5070
+- **Role:** admin
+
+## Session Summary
+
+Two threads on Glaztech today: (1) a customer-facing website outage that was diagnosed and resolved, and (2) scoping the least-privilege SQL login migration (#32378 / coord todo `aebaf751`), which turned into a significant infrastructure-recon effort that materially changed the understanding of the environment.
+
+**Outage:** `glaztech.com` (WWW, 192.168.8.72 / 65.113.52.88) returned site-wide HTTP 500 then 401. Root cause: a 2026-06-04 change applied the E1 hardening (removed the insecure `Everyone:(R)` from the web root `D:\web\glaztech_4`) but did not restore read access for the IIS serving accounts — and this server's `IIS_IUSRS` group was non-default, missing `IUSR` (the anonymous-auth identity). The IIS worker could load the app (DLLs + Web.config had explicit grants) but could not read the ~43,600-file content tree → `500.50` then `401.3` (access-denied-by-ACL). A web.config rollback was a red herring (preserved as `Web.config.broken-20260605-084124`). Fix (least-privilege; `Everyone` stays removed — completing E1 correctly): added `IUSR` + `IIS APPPOOL\glaztech_new` to `IIS_IUSRS`, granted `IIS_IUSRS` ReadAndExecute across the content tree (detached `icacls` — slow on 43.6K files, exceeded the agent command timeout), then `iisreset` to regenerate the cached IUSR token (an app-pool recycle was insufficient). Verified: last 60 requests 55x 200 / 0x 401 / 0x 500; external 200s on apex, www, customer_login.aspx, and product images. Posted a customer-visible #32378 comment ("Website service restored"), then resent it after Mike updated the ticket contacts (Tom primary, Alex + Steve CC).
+
+**Least-privilege scope:** drafted a full scope for replacing the website's `sysadmin` login `tom` with a least-privilege login. The central constraint is GTIware co-residency (in-process card-engine DLLs share the website's `Web.config` connection and need `cc_file`), forcing a two-phase plan (Phase 1 strip sysadmin + the OS-RCE/domain-admin/cross-office/linked-server blast radius, keep `cc_file`; Phase 2 DENY `cc_file` after GTIware is decoupled). Routed the scope to **Grok 4.3 and Gemini 3 Pro** for independent adversarial review — both CONCUR with the two-phase approach and the matrix, with no material disagreement. Corrections folded in: don't DENY `tempdb`; audit linked-login catch-all mappings; dynamic SQL breaks ownership chaining; app-pool recycle to flush the ADO.NET pool on swap and rollback; alphanumeric-only interim password; fixed a rotation-gating contradiction; don't overstate Phase 1 (containment, not card-safety).
+
+**Recon — the picture changed:** Mike suspected per-site SQL servers across a meshed 192.168.0-9.x network. Read-only RMM recon (GuruRMM agents on WWW + GTI-INV-SQL) showed it's **centralized now** (his memory matches the OLD topology, preserved in commented-out Web.config strings). All office data lives on one instance `GTI-INV-SQL\GTISQL` (192.168.8.62,**3436**, SQL 2012) — ~57 per-office DBs; a separate default instance (2008 R2) holds payroll/reporting; a third instance is on `:3430`. The website's `Web.config` revealed its **true footprint**: ~15 connection strings, all as `tom`, reaching **all 10 offices' `glaz_prod_*` DBs + Sage accounting (`mas_gti` @ 192.168.0.55) + payroll (`qqest`) + `msdb`**. That contradicts the scope's "DENY other offices/accounting/payroll/msdb" — the app legitimately uses them, shrinking Phase 1's value and favoring the architectural fixes. Mike parked the migration pending a real network recon (he's enrolling the site servers in RMM) and saved everything.
+
+## Key Decisions
+
+- **Outage fix completed E1 correctly** (least-privilege ACL, `Everyone` stays removed) rather than reverting Tom's hardening — turned the incident into a remediation step done right.
+- **`iisreset` over app-pool recycle** — the anonymous `IUSR` token is cached at the IIS service level; only a full reset picks up the new `IIS_IUSRS` membership.
+- **Detached `icacls`** to escape the agent command timeout on the 43.6K-file ACL propagation.
+- **No blame in the ticket comment** (Mike's instruction) — described only what was done; customer-visible + emailed to the updated contacts.
+- **Two-phase least-privilege migration** forced by GTIware co-residency; routed to two independent models for concurrence before treating it as execution-ready.
+- **Parked the migration** once recon showed the website is a cross-office + accounting + payroll + msdb hub on one sysadmin credential — a clean least-priv login barely exists; the durable fix is architectural, and a full network recon is the prerequisite.
+
+## Problems Encountered
+
+- `icacls /T` on 43,602 files exceeded the agent command timeout repeatedly → ran it detached (Start-Process) and polled a completion marker.
+- App-pool recycle didn't clear `401.3` → root cause was the cached IUSR token; `iisreset` resolved it.
+- `web.config` rollback didn't fix the outage (it was an ACL problem, not config) — preserved Tom's file rather than discarding his hardening.
+- Recon hit the wrong SQL instance first (default instance via `localhost -E`); the cards instance is the named `GTISQL` on `:3436`.
+- `SYSTEM` (agent) is **not** sysadmin on `:3436`, so the authoritative login/role/`tom` map there still needs the `tom` credential or a sysadmin Windows login.
+
+## Configuration Changes
+
+**Glaztech WWW (192.168.8.72) — production server, via GuruRMM:**
+- `IIS_IUSRS` local group: added `NT AUTHORITY\IUSR` and `IIS APPPOOL\glaztech_new` (were missing — non-default).
+- `icacls "D:\web\glaztech_4" /grant "IIS_IUSRS:(OI)(CI)(RX)" /T` — granted app-pool/anonymous read across the content tree (completes E1; `Everyone:(R)` remains removed).
+- `iisreset /restart`.
+- `Web.config` rolled back to `Web.config.bak-20260604-170500` (the working 6/3 version); Tom's 6/4 edit preserved as `Web.config.broken-20260605-084124` (NOT the cause; contains debug=false + security headers + secure cookies, for re-apply after coordinating with Tom).
+
+**Repo:**
+- Created `clients/glaztech/reports/2026-06-05-least-privilege-db-migration-scope.md` (v0.3 — scope + Grok/Gemini review + recon findings; PARKED).
+- Created this session log.
+
+**Syncro:**
+- #32378 comment 417493519 ("Website service restored", customer-visible) + resend 417494988 to updated contacts.
+
+## Infrastructure & Servers (Glaztech SQL topology — corrected 2026-06-05)
+
+- **GTI-INV-SQL** (machine, 192.168.8.62, at "INV - Involta" colo) runs **3 SQL instances**:
+  - **Default instance** — SQL Server **2008 R2** (10.50.2550), DBs: `qqest`, ReportServer(+TempDB), system. `NT AUTHORITY\SYSTEM` IS sysadmin here.
+  - **`GTI-INV-SQL\GTISQL` on port 3436** — SQL Server **2012** (11.0.7507) — **the cards/website instance**, ~57 DBs: `glaz_prod_<office>` + `_archive` + `_web` for **alb, boi, brl, corp, den, elp, phx, shp, slc, tuc**, PDF stores (`glaz_pdf*`), `gti_samsara`, `qqest`. `xp_cmdshell=1`. `SYSTEM` is NOT sysadmin here.
+  - **Third instance on 192.168.8.62,3430.**
+- **Linked servers (from `:3436`):** 192.168.0.54,55181 · 192.168.0.55,55181 (`mas_gti`/Sage) · 192.168.8.52,3436 (backup) · 192.168.8.212,3436 (backup) · 192.168.8.62,3430 · `GLAZ\TIMEFORCE` (`qqest`). All data-access enabled; default-instance linked logins map `(default-all) → tom`.
+- **Website (`WWW`) connection strings (active, all `user id=tom`):** glaz_prod (tuc), glaz_prod_phx/_slc/_elp/_den/_alb/_boi/_brl/_shp/_corp, glaz_pdf, glaz_pdf_corp — all on 192.168.8.62,3436; **`mas_gti` @ 192.168.0.55,55181** (Sage); **`qqest` via glaz\timeforce** (payroll); **`msdb` @ 192.168.8.62,3436** (`glaztech_jobs`).
+- **Old (commented-out) topology** (matches Mike's per-site memory): per-office ports `glaz,3430` (tuc) / 3432 (phx) / 3438 (slc) / 3431 (elp) / 3435 (den) / 3433 (alb) / 3437 (boi) / 3439 (brl); `sql1,3436`; `sql3,3430`. Since consolidated.
+- **`tom`** SQL login created 2017-12-30; sysadmin; cross-mesh remote login. Password in WWW `Web.config` (cleartext, NOT vaulted; redacted from all artifacts).
+- **GuruRMM Glaztech agents (4):** WWW (455a1bc7), GTI-INV-SQL (869e56b4), GTI-INV-DC, GTI-INV-DC1.
+
+## Commands & Outputs (key)
+
+```
+# Outage fix (WWW, via /rmm)
+net localgroup IIS_IUSRS IUSR /add ; net localgroup IIS_IUSRS "IIS APPPOOL\glaztech_new" /add
+icacls "D:\web\glaztech_4" /grant "IIS_IUSRS:(OI)(CI)(RX)" /T /C /Q   # 43,602 files; run detached
+iisreset /restart
+# verify: last 60 IIS requests -> 55x200 0x401 0x500 ; external 200 on apex/www/login/images
+
+# SQL recon (GTI-INV-SQL, via /rmm, sqlcmd -S localhost[,3436] -E, read-only)
+SELECT name FROM sys.databases                  # default inst: qqest/ReportServer/system ; :3436: ~57 glaz_prod_* etc
+SELECT name,data_source FROM sys.servers WHERE is_linked=1   # the mesh (0.54/0.55/8.52/8.212/8.62:3430/timeforce)
+# Web.config connectionStrings on WWW -> all user id=tom across all offices + mas_gti + qqest + msdb
+```
+
+## Pending / Incomplete Tasks
+
+- **Least-privilege `tom` migration — PARKED** pending a **full network recon** (Mike enrolling site servers in RMM) to map how the estate fits together; the website's true footprint (all offices + accounting + payroll + msdb) makes a clean least-priv login barely viable → reconsider in favor of the architectural fixes (assessment items 16-17, 22). Scope: `clients/glaztech/reports/2026-06-05-least-privilege-db-migration-scope.md` (v0.3). Coord todo `aebaf751`.
+- **Authoritative `:3436` login/role recon** still needs the `tom` credential (or a sysadmin Windows login) — `SYSTEM` isn't sysadmin there.
+- **Emergency containment (E2-E5)** still open and now more urgent: E2 (rotate `glaztech\administrator` + strip cleartext from `msdb` job steps) is a hard prerequisite because the website legitimately holds an `msdb` connection. E3 (disable `xp_cmdshell` — still =1 on `:3436`), E4 (de-priv SQL Agent / disable `sa`), E5 (firewall/RealVNC).
+- **Tom's `web.config` hardening** (debug=false + security headers + secure cookies) preserved on WWW; re-apply after coordinating with Tom.
+- Whole web-app remediation remains **parked on #32378 (Waiting on Customer)**.
+- Failed-login detection/lockout (H5): options laid out; waiting on Tom (app-side) — no evidence of active attack.
+
+## Reference Information
+
+- Scope doc: `clients/glaztech/reports/2026-06-05-least-privilege-db-migration-scope.md`
+- Assessment: `clients/glaztech/reports/2026-06-03-website-security-assessment.md`
+- Ticket: #32378 (id 112111185), Waiting on Customer. Comments 417493519 + 417494988.
+- Coord todos: `aebaf751` (least-priv `tom` migration), `6d15fc88` (E2-E4 containment).
+- GuruRMM: WWW agent `455a1bc7-1c29-42bc-b597-fa1e64f08eec`; GTI-INV-SQL agent `869e56b4-e8ed-4808-8c88-782d1577c152`.