glaztech: least-privilege tom DB migration scope + 2026-06-05 session log
Scope (v0.3) for replacing the website's sysadmin login 'tom' with a least-privilege login: two-phase plan (GTIware co-residency forces keeping cc_file in Phase 1), Grok + Gemini independent review folded in, and live RMM recon findings that materially changed the picture - the website is a cross-office + Sage accounting + payroll + msdb hub on one sysadmin credential, SQL is centralized on GTI-INV-SQL\GTISQL:3436 (not per-site). PARKED pending a full network recon. Session log covers the website outage fix (incomplete E1 ACL hardening) + the scoping + recon. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,184 @@
|
||||
# Glaztech — Least-Privilege DB Login Migration: Full Scope
|
||||
|
||||
**Status:** DRAFT v0.3 — PARKED pending full network recon (see §11). Grok + Gemini reviewed; **live RMM recon (§11) materially changed the picture — §1-§3 are too optimistic and must be reconciled before execution.** · **Date:** 2026-06-05
|
||||
**Owner:** ACG · **Ticket:** #32378 (Waiting on Customer) · **Coord todo:** `aebaf751`
|
||||
**Source of truth:** `clients/glaztech/reports/2026-06-03-website-security-assessment.md`
|
||||
(C0, C0-Extended, H6, and the "Sequencing the least-privilege DB migration" callout)
|
||||
|
||||
> Objective: stop the internet-facing website (`D:\web\glaztech_4`, IIS site `glaztech_new`,
|
||||
> server `WWW` 192.168.8.72 / 65.113.52.88) from connecting to `GTI-INV-SQL`
|
||||
> (192.168.8.62,3436) as the `sysadmin` login `tom`, replacing it with a least-privilege
|
||||
> login — without taking the live payment site or the card-charging engine down.
|
||||
|
||||
---
|
||||
|
||||
## 1. The central constraint that shapes everything: GTIware co-residency
|
||||
|
||||
The website's `Web.config` `tom` connection is **shared, in the same IIS worker process**, with the GTIware card-on-file engine DLLs (`gt_auto_process_2020.dll`, `glaztech_utilities_2020.dll` in `D:\web\glaztech_4\Bin\`). Those DLLs **carry no connection strings of their own** — they use the website's `tom` connection — and they legitimately need `cc_file` / `cof_payments_header` (card tables) to write and charge saved cards.
|
||||
|
||||
**Consequence:** a *true* least-privilege website login (with `DENY` on `cc_file`) **cannot be applied while the card engine shares that connection** — it would break auto-billing. Per assessment item 22, "you cannot safely swap only the website's connection string while the engine shares the same process and config."
|
||||
|
||||
This forces a **two-phase** migration:
|
||||
|
||||
- **Phase 1 (achievable now, ACG-drivable):** remove the catastrophic rights — `sysadmin`/`securityadmin`/`dbcreator`, cross-office DB reach, `*_archive`, `msdb`/`master`, linked servers, and the `xp_cmdshell` OS-RCE path — while **retaining** the site's own-office DB access **and** `cc_file`/`cof_payments_header` (because the in-process engine still needs them). This alone collapses the worst of C0: SQLi can no longer reach OS-RCE, the domain-admin password in `msdb`, other offices' data, or the linked-server mesh.
|
||||
- **Phase 2 (gated on Tom's parallel GTIware decoupling — item 17/22):** once the card engine runs off its own dedicated connection/host, `DENY cc_file`/`cof_payments_header` on the website login, completing true least-privilege.
|
||||
|
||||
**Phase 1 delivers ~80% of the risk reduction without depending on Tom's code** — it collapses the catastrophic chain (OS-RCE via `xp_cmdshell`, the cleartext domain-admin password in `msdb`, all *other* offices + archives, the linked-server pivot, raw `web_security` dumps). **But be precise (both models): Phase 1 is "blast radius contained," NOT "cards safe."** Because the web login still holds `cc_file`/`cof_payments_header` GRANT for the engine, a SQLi (C3) plus a guessed customer login still yields full PAN+CVV **for the office this portal serves**. That residual card exposure closes only in Phase 2 (DENY `cc_file`) after the GTIware decoupling — and ultimately in the tokenization/purge workstream (assessment items 16-17).
|
||||
|
||||
---
|
||||
|
||||
## 2. Pre-execution inventory (callout step 1 — MUST complete before any change)
|
||||
|
||||
Two inventories, both **read-only**. These are the gating tasks; do not draft the grant set until they're done.
|
||||
|
||||
### 2a. What objects the website actually touches (to build the GRANT set)
|
||||
- Source-side: enumerate every table/view/stored-proc referenced by the site's SQL — the 948 parameterized calls + 59 concatenated statements. Grep the VB.NET source on the box (`D:\web\glaztech_4`, excluding `Old_bin`/`Old_code`) for `SqlCommand`, `.CommandText`, proc names, `FROM`/`JOIN`/`INTO`/`UPDATE`/`EXEC` targets; dedupe to an object list per database.
|
||||
- DB-side cross-check: SQL Server **Extended Events / Profiler trace** on the `tom` login over a representative window (a full business day incl. an `gt_auto_process` run) to capture the actual object-access set empirically — catches dynamic SQL the static grep misses. (Read-only; trace metadata only.)
|
||||
|
||||
### 2b. Every consumer of the `tom` login (to plan the password rotation, callout step 5)
|
||||
Rotating `tom` without updating every consumer in the same change window breaks live systems. Enumerate:
|
||||
- **Website** `Web.config` connection strings (primary).
|
||||
- **In-process GTIware DLLs** — confirmed users of the website connection.
|
||||
- **Server-side billing jobs:** `d:\sql_jobs\bin\gt_console_apps.exe` (modes cp/is/oa/lo) run by SQL Agent — confirm whether it authenticates as `tom` or the Agent (domain-admin) account.
|
||||
- **SQL Agent jobs / job steps** referencing `tom` (`msdb.dbo.sysjobsteps`).
|
||||
- **Linked servers** whose remote-login mapping uses `tom` (`sys.servers` + `sys.linked_logins`).
|
||||
- **The second host** `\\192.168.0.147\web\glaztech_4` (same code tree — may hold its own `tom` Web.config).
|
||||
- **`/webhooks` + `/webhooks1`** (Samsara) apps — do they carry a `tom` connection?
|
||||
- Any scheduled tasks / internal apps / Excel/Access ODBC DSNs using `tom`.
|
||||
Queries: `sys.dm_exec_sessions`/`sys.dm_exec_connections` filtered by `login_name='tom'` sampled over time; `sys.linked_logins`; `msdb` job-step scan.
|
||||
|
||||
---
|
||||
|
||||
## 3. Phase 1 — de-privilege (the achievable-now core)
|
||||
|
||||
### 3a. New login + grant/DENY matrix (draft — finalize after §2a)
|
||||
|
||||
Create a **new SQL login** (e.g. `web_glaztech_app`) — SQL auth, **no server roles** beyond `public`. Use a **high-entropy but alphanumeric-only password** (avoid `; ' = + " { }`), vaulted — legacy VB.NET/GTIware connection-string parsing may mis-handle special characters and silently truncate/fail the string (Gemini).
|
||||
|
||||
| Scope | Action |
|
||||
|---|---|
|
||||
| Own-office production DB(s) the site serves (`glaz_prod`, `glaz_prod_tuc`, etc. — per §2a) | `GRANT` EXECUTE on the site's procs + SELECT/INSERT/UPDATE on the specific tables/views it uses (least-priv, object-level, **not** `db_owner`) |
|
||||
| `cc_file`, `cof_payments_header` (card tables) | **Phase 1: GRANT** (in-process engine needs them) → **Phase 2: DENY** after GTIware decoupling |
|
||||
| `web_security` (passwords) | GRANT EXECUTE on `get_web_accesslevel` only; **DENY** direct SELECT on the table. Works via **ownership chaining** (proc reads the table on the caller's behalf; raw `SELECT * FROM web_security` injection blocked) — **only if** the proc is `dbo`-owned, has **no dynamic SQL** (`EXEC(@sql)`/`sp_executesql`) and **no `EXECUTE AS`**. Dynamic SQL runs under the caller and would hit the DENY. *Scan the proc set (§2a) for dynamic SQL and explicitly GRANT any tables reached that way* (both models). |
|
||||
| Every **other office's** DB + all `*_archive` DBs | **DENY** (no cross-DB reach). **But first confirm no legitimate view/proc/report joins current data with archive/other-office data** — cross-DB ownership chaining is OFF by default, so such a join will hard-break under the DENY. The §2a trace must surface cross-DB dependencies (Gemini). |
|
||||
| `master`, `msdb`, `model` | **DENY**; **no** `xp_cmdshell` (login isn't sysadmin, so it can't enable/run it). **Do NOT `DENY tempdb`** — `public` must keep local `#temp` creation, which legacy procs/the engine likely use; a tempdb DENY breaks them on swap. (Watch for `##global` temp / physical tempdb tables in legacy code — those would still fail.) (both models) |
|
||||
| Linked servers (all 7) | No grants. **Do NOT assume non-sysadmin = blocked** — audit `sys.servers` + `sys.linked_logins` for a catch-all *"connections made using this security context"* mapping to a remote `sa`/sysadmin; the new login would silently inherit those rights on the remote box. Set the per-login mapping to *"Not be made"* and test from the new login during the window. (both models) |
|
||||
| Server roles | none (`public` only) — removes `sysadmin`/`securityadmin`/`dbcreator` reach |
|
||||
|
||||
**Net Phase-1 effect:** a SQLi via `quo()` now executes as a constrained login — no OS RCE, no `msdb` domain-admin password, no other-office/archive data, no linked-server pivot, no raw password-table dump. Card tables remain reachable *only* because the co-resident engine requires them (closed in Phase 2).
|
||||
|
||||
### 3b. Sequence (callout steps 2-4)
|
||||
1. Create `web_glaztech_app` + apply the grant/DENY matrix (no impact yet — nothing uses it).
|
||||
2. **Staged `Web.config` swap** to the new login, in a maintenance window, with **SQL + app error monitoring live** (SQL error log, IIS/app logs, the new failed-login logging if available). Watch a full cycle including an `gt_auto_process` run.
|
||||
3. Any missing object permission surfaces as a SQL error → grant it (re-validate §2a) or roll back.
|
||||
4. Once proven in production across a representative window, **revoke `sysadmin`/`securityadmin`/`dbcreator` from `tom`** (only after confirming no *other* consumer needs those rights — §2b).
|
||||
|
||||
**Connection-pool note (Gemini):** both the swap (step 2) **and** the rollback **must** include an IIS app-pool **recycle**. A `Web.config` edit alone does not flush ADO.NET's existing pooled `tom` connections — lingering pooled connections can mask the new login's failures or cause inconsistent behavior. Recycle on every connection-string change so the worker opens fresh connections under the intended login.
|
||||
|
||||
**Both website and engine move at once:** the in-process GTIware DLLs read the **same** `Web.config`, so the swap moves the website *and* the card engine to `web_glaztech_app` simultaneously. Phase 1's `cc_file`/`cof_payments_header` GRANT (and every other engine-touched object found in §2a) must be in place before the swap, or auto-billing fails the instant the pool recycles. (First confirm the DLLs truly source the string from `Web.config` and don't hardcode `tom` — §2b / Q in §8.)
|
||||
|
||||
### 3c. Rollback
|
||||
Single-step, fast: revert the `Web.config` connection string to `tom` and **recycle the app pool** (required to flush the pool — see note above). Keep `tom` intact and unchanged until Phase 1 is proven — do **not** demote/rotate `tom` until step 4/5. Take a `Web.config` backup before each edit (those backups still contain the old plaintext `tom` password — pair this work with E1 so they aren't world-readable).
|
||||
|
||||
---
|
||||
|
||||
## 4. `tom` password rotation (callout step 5 — removes the plaintext sysadmin cred)
|
||||
|
||||
**Corrected per Gemini's push-back (the earlier draft was self-contradictory).** Because the in-process card engine reads the **same `Web.config`**, the Phase 1 swap **vacates `tom` entirely from the IIS worker** — both the website and the engine are now on `web_glaztech_app`. So rotation is **NOT** gated on Phase 2 or the engine; it is gated only on the **external** `tom` consumers from §2b (the `.147` host, SQL Agent jobs, `gt_console_apps.exe`, ODBC/Excel DSNs, linked-server remote mappings).
|
||||
|
||||
Sequence: after Phase 1 is proven AND §2b is complete, rotate `tom`'s password and update **every** external consumer **in the same change window**. (If §2b reveals the engine does **not** in fact read `Web.config` but hardcodes/uses a separate `tom` string, then §1's shared-connection premise is wrong and the whole plan needs revisiting — so confirming the DLLs' connection source in §2b is a hard prerequisite, not a detail.)
|
||||
|
||||
---
|
||||
|
||||
## 5. Phase 2 — remove card-table access (gated on GTIware decoupling)
|
||||
|
||||
Dependency: Tom's parallel GTIware workstream (assessment items 17 + 22) gives the card engine its own dedicated low-privilege connection/host. Then: `DENY cc_file`, `cof_payments_header` on `web_glaztech_app`; re-validate the website (which per assessment stores no cards itself) still functions; confirm auto-billing runs on the engine's own login.
|
||||
|
||||
---
|
||||
|
||||
## 6. Environment, change window, testing gates
|
||||
|
||||
- **No staging assumed (H1 — dev tooling + source on the live box).** Plan a **maintenance window with immediate rollback**, or stand up a minimal validation copy first. **No in-place hotfixes without a rollback path.**
|
||||
- Validate per the assessment's testing gates: DB-login change reverts via `Web.config`; per-page validation that queries still return correct results after the swap; confirm `gt_auto_process` auto-charge still runs on the new login.
|
||||
- **Take a backup-first posture** on any object whose permissions change; never touch `tom` until the replacement is proven.
|
||||
|
||||
## 7. Risks & mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Missed object permission → site error after swap | Empirical XEvents trace (§2a) + live error monitoring during the staged swap + one-step rollback |
|
||||
| Card engine breaks (loses `cc_file`) | Phase 1 retains `cc_file`; removal only in Phase 2 after decoupling |
|
||||
| Rotating `tom` breaks an un-inventoried consumer | Full §2b inventory + same-window update; defer rotation if any consumer is unresolved |
|
||||
| No staging → prod-only change | Maintenance window + immediate `Web.config` rollback; backups |
|
||||
| `corp` DB `cc_file` "Invalid object name" anomaly | Resolve before finalizing grants (open follow-up) |
|
||||
| Second host `192.168.0.147` has its own `tom` Web.config | Include in §2b; migrate/rotate in the same window |
|
||||
| **Legit cross-DB join** (current × archive/other-office) hard-breaks under DENY | Surface cross-DB dependencies in the §2a trace before applying the DENYs |
|
||||
| **Dynamic SQL in a proc** breaks ownership chaining → hits DENY | Scan procs for `EXEC`/`sp_executesql`; GRANT tables they reach dynamically |
|
||||
| **ADO.NET pool not flushed** → stale `tom` connections mask failures | App-pool recycle on every connection-string change (swap and rollback) |
|
||||
| **Connection-string parser** chokes on special chars in the new password | Alphanumeric-only high-entropy password for the interim login |
|
||||
| **`tempdb` DENY** breaks `#temp` creation | Do not DENY `tempdb`; `public` keeps local temp; watch `##global`/physical tempdb use |
|
||||
| **Linked-server catch-all mapping** to remote `sa` | Audit `sys.linked_logins`; set "Not be made" for the new login; test |
|
||||
| `gt_console_apps.exe` (server-side charging) opens its own `tom` connection | Confirm in §2b; if so it's an external consumer for the rotation window |
|
||||
|
||||
## 8. Open questions for Tom / Steve
|
||||
|
||||
1. Confirm the website reads `web_security` **only** via `get_web_accesslevel` (so we can DENY direct table SELECT).
|
||||
2. Does `gt_auto_process` re-submit CVV at charge time? (affects the parallel CVV purge, not this migration, but same window).
|
||||
3. Will GTIware get its own DB connection/host (Phase 2 prerequisite)? Timeline?
|
||||
4. Is there *any* test/validation environment, or is the maintenance-window-on-prod path required?
|
||||
5. Authoritative list of `tom` consumers beyond the website (jobs, the `.147` host, Samsara webhooks, ODBC DSNs)?
|
||||
|
||||
## 9. Effort / sequencing summary
|
||||
|
||||
- **Inventory (§2):** read-only; ~prerequisite, do first.
|
||||
- **Phase 1 (§3):** ACG-drivable with client sign-off + a maintenance window; the high-value de-sysadmin step. Reversible.
|
||||
- **`tom` rotation (§4):** after Phase 1 proven + consumer inventory complete.
|
||||
- **Phase 2 (§5):** gated on Tom's GTIware decoupling (parallel workstream).
|
||||
|
||||
---
|
||||
|
||||
## 10. Independent review (Grok 4.3 + Gemini 3 Pro, 2026-06-05)
|
||||
|
||||
Both models reviewed this scope and the underlying assessment independently and **CONCUR** with the two-phase approach, the grant/DENY matrix, and the inventory-first / prove-before-touching-`tom` posture. They **agreed** on every correction now folded in above:
|
||||
|
||||
- Two-phase keeping `cc_file` in Phase 1 is *required* (not just safer) given the in-process engine; no safer pure-interim exists without moving the engine first.
|
||||
- **Do not DENY `tempdb`** (breaks `#temp`); **don't assume** non-sysadmin blocks linked servers (audit catch-all mappings); **dynamic SQL** in procs breaks ownership chaining (scan + GRANT); the empirical **XEvents trace is the true gate**.
|
||||
- Confirmed: losing sysadmin reliably removes `xp_cmdshell`; DB-level `DENY` overrides role GRANTs; the `web_security` EXECUTE-only + DENY pattern works via ownership chaining (dbo-owned, no dynamic SQL, no `EXECUTE AS`).
|
||||
- **Gemini's unique catches:** the §4 rotation-gating contradiction (now fixed — `tom` is vacated from the worker at the Phase 1 swap, so rotation gates on *external* consumers only); the **ADO.NET pool flush / app-pool recycle** requirement on swap *and* rollback; the **alphanumeric-only password** to avoid legacy connection-string parser failures; cross-DB-join breakage under DENY.
|
||||
- **Grok's unique catches:** don't overstate Phase 1 ("contained," not "cards safe" — served-office cards still exposed via SQLi until Phase 2); proc-level grep for `cc_file` before the Phase 2 DENY; confirm `gt_console_apps.exe`'s connection source; pair the swap with E1 (old `tom` password lingers in `Web.config` backups).
|
||||
|
||||
No disagreement between the two models on any material point — a strong signal the plan is sound, contingent on the §2 inventories.
|
||||
|
||||
---
|
||||
|
||||
## 11. Recon findings (live, via GuruRMM read-only, 2026-06-05) — MATERIAL changes
|
||||
|
||||
Read-only SQL + `Web.config` recon (no data modified) corrected several core assumptions. **These supersede parts of §1-§3 and must be reconciled before any execution.**
|
||||
|
||||
### Topology — centralized now; per-site SQL was the OLD layout
|
||||
- `GTI-INV-SQL` (at the **"INV - Involta"** colo, 192.168.8.62) runs **multiple instances**: a **default instance** (SQL **2008 R2** / 10.50, EOL) with only `qqest` + ReportServer + system; the named instance **`GTI-INV-SQL\GTISQL` on port 3436** (SQL **2012** / 11.0, EOL) holding **all office data** (~57 DBs: `glaz_prod_<office>`, `_archive`, `_web` for alb/boi/brl/corp/den/elp/phx/shp/slc/tuc + PDF stores + `gti_samsara` + `qqest`); and a **third** instance on `192.168.8.62,3430`.
|
||||
- **The per-site-SQL memory matches the OLD topology** — the **commented-out** Web.config strings show per-office ports/instances (`glaz,3430` tuc … `glaz,3439` brl, `sql1,3436`, `sql3,3430`). It has since been **consolidated** onto the single `:3436` instance with per-office *databases*.
|
||||
- **The "mesh" = a handful of linked servers** on `:3436` (192.168.0.54; 192.168.0.55 = Sage `mas_gti`; 192.168.8.52 + .212 = backups; 192.168.8.62,3430; `GLAZ\TIMEFORCE`), all data-access enabled. On the default instance every linked-login maps `(default-all) → tom` — **`tom` is the cross-system credential.** NOT one-SQL-per-office.
|
||||
|
||||
### The website's TRUE footprint (Web.config, ~15 active connection strings — all `user id=tom`, all on `:3436` unless noted)
|
||||
The site is **not** single-office. As sysadmin `tom` it legitimately connects to:
|
||||
- **All 10 offices' production DBs** (`glaz_prod`, `glaz_prod_phx/_slc/_elp/_den/_alb/_boi/_brl/_shp/_corp`) + PDF stores (`glaz_pdf`, `glaz_pdf_corp`)
|
||||
- **Sage accounting** — `mas_gti` @ `192.168.0.55,55181` (DIRECT connection, not just linked)
|
||||
- **Payroll** — `qqest` via `glaz\timeforce`
|
||||
- **`msdb`** — `glaztech_jobs` → `192.168.8.62,3436/msdb` (the DB holding the cleartext domain-admin password + Agent jobs)
|
||||
- Uses the **operational `glaz_prod_*` DBs directly** — NOT the `_web` databases (no `_web` simplification).
|
||||
|
||||
### Implications — §1-§3 are too optimistic
|
||||
- **§3's "scope to own-office DB; DENY other offices/archives/msdb/accounting/payroll" is contradicted** — the app legitimately uses all of them. A least-privilege login can't DENY what the app connects to.
|
||||
- **Phase 1's blast-radius win shrinks:** removing sysadmin still kills `xp_cmdshell` RCE + arbitrary-instance control, but the new login would still legitimately reach all offices, accounting, payroll, and `msdb` — most of the estate. Retaining `msdb` means the cleartext domain-admin password in `sysjobsteps` stays reachable unless **E2 (rotate it) is done first** — so E2 is now a hard prerequisite, and we must learn WHAT the website does in `msdb`.
|
||||
- **The durable fix is architectural** (assessment items 16-17, 22): separate the website's data + decouple — as-built the website is a cross-office hub wired to accounting + payroll + msdb under one sysadmin credential, so a "clean least-privilege login" barely exists.
|
||||
- **EOL SQL** (2008 R2 + 2012) adds platform risk and limits options.
|
||||
|
||||
### Recon limits / still needed
|
||||
- `SYSTEM` (the agent) is sysadmin on the **default** instance but **NOT** on `:3436` — the authoritative `tom`-role / sysadmin-login / linked-login map on `:3436` needs the `tom` credential (or a sysadmin Windows login).
|
||||
- **Full network recon PENDING** — Mike is enrolling the site servers in RMM to confirm no per-site SQL remains and map how accounting/payroll/backups/the 0.x + 8.x subnets fit together.
|
||||
- The §2a object-inventory (XEvents trace) is even more important given the multi-DB, cross-server footprint.
|
||||
|
||||
---
|
||||
|
||||
*v0.3 — Grok + Gemini corrections incorporated (§10); live RMM recon findings added (§11). **PARKED** by Mike 2026-06-05: the migration is larger than the assessment framed (website is a multi-office + accounting + payroll + msdb hub on one sysadmin credential), and a real network recon is needed to understand how the infrastructure fits together before this can be safely executed. Also still pending Tom/Steve answers to §8. Whole web-app remediation remains parked on #32378.*
|
||||
92
clients/glaztech/session-logs/2026-06-05-session.md
Normal file
92
clients/glaztech/session-logs/2026-06-05-session.md
Normal file
@@ -0,0 +1,92 @@
|
||||
# Glaztech Session Log — 2026-06-05
|
||||
|
||||
## User
|
||||
- **User:** Mike Swanson (mike)
|
||||
- **Machine:** GURU-5070
|
||||
- **Role:** admin
|
||||
|
||||
## Session Summary
|
||||
|
||||
Two threads on Glaztech today: (1) a customer-facing website outage that was diagnosed and resolved, and (2) scoping the least-privilege SQL login migration (#32378 / coord todo `aebaf751`), which turned into a significant infrastructure-recon effort that materially changed the understanding of the environment.
|
||||
|
||||
**Outage:** `glaztech.com` (WWW, 192.168.8.72 / 65.113.52.88) returned site-wide HTTP 500 then 401. Root cause: a 2026-06-04 change applied the E1 hardening (removed the insecure `Everyone:(R)` from the web root `D:\web\glaztech_4`) but did not restore read access for the IIS serving accounts — and this server's `IIS_IUSRS` group was non-default, missing `IUSR` (the anonymous-auth identity). The IIS worker could load the app (DLLs + Web.config had explicit grants) but could not read the ~43,600-file content tree → `500.50` then `401.3` (access-denied-by-ACL). A web.config rollback was a red herring (preserved as `Web.config.broken-20260605-084124`). Fix (least-privilege; `Everyone` stays removed — completing E1 correctly): added `IUSR` + `IIS APPPOOL\glaztech_new` to `IIS_IUSRS`, granted `IIS_IUSRS` ReadAndExecute across the content tree (detached `icacls` — slow on 43.6K files, exceeded the agent command timeout), then `iisreset` to regenerate the cached IUSR token (an app-pool recycle was insufficient). Verified: last 60 requests 55x 200 / 0x 401 / 0x 500; external 200s on apex, www, customer_login.aspx, and product images. Posted a customer-visible #32378 comment ("Website service restored"), then resent it after Mike updated the ticket contacts (Tom primary, Alex + Steve CC).
|
||||
|
||||
**Least-privilege scope:** drafted a full scope for replacing the website's `sysadmin` login `tom` with a least-privilege login. The central constraint is GTIware co-residency (in-process card-engine DLLs share the website's `Web.config` connection and need `cc_file`), forcing a two-phase plan (Phase 1 strip sysadmin + the OS-RCE/domain-admin/cross-office/linked-server blast radius, keep `cc_file`; Phase 2 DENY `cc_file` after GTIware is decoupled). Routed the scope to **Grok 4.3 and Gemini 3 Pro** for independent adversarial review — both CONCUR with the two-phase approach and the matrix, with no material disagreement. Corrections folded in: don't DENY `tempdb`; audit linked-login catch-all mappings; dynamic SQL breaks ownership chaining; app-pool recycle to flush the ADO.NET pool on swap and rollback; alphanumeric-only interim password; fixed a rotation-gating contradiction; don't overstate Phase 1 (containment, not card-safety).
|
||||
|
||||
**Recon — the picture changed:** Mike suspected per-site SQL servers across a meshed 192.168.0-9.x network. Read-only RMM recon (GuruRMM agents on WWW + GTI-INV-SQL) showed it's **centralized now** (his memory matches the OLD topology, preserved in commented-out Web.config strings). All office data lives on one instance `GTI-INV-SQL\GTISQL` (192.168.8.62,**3436**, SQL 2012) — ~57 per-office DBs; a separate default instance (2008 R2) holds payroll/reporting; a third instance is on `:3430`. The website's `Web.config` revealed its **true footprint**: ~15 connection strings, all as `tom`, reaching **all 10 offices' `glaz_prod_*` DBs + Sage accounting (`mas_gti` @ 192.168.0.55) + payroll (`qqest`) + `msdb`**. That contradicts the scope's "DENY other offices/accounting/payroll/msdb" — the app legitimately uses them, shrinking Phase 1's value and favoring the architectural fixes. Mike parked the migration pending a real network recon (he's enrolling the site servers in RMM) and saved everything.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- **Outage fix completed E1 correctly** (least-privilege ACL, `Everyone` stays removed) rather than reverting Tom's hardening — turned the incident into a remediation step done right.
|
||||
- **`iisreset` over app-pool recycle** — the anonymous `IUSR` token is cached at the IIS service level; only a full reset picks up the new `IIS_IUSRS` membership.
|
||||
- **Detached `icacls`** to escape the agent command timeout on the 43.6K-file ACL propagation.
|
||||
- **No blame in the ticket comment** (Mike's instruction) — described only what was done; customer-visible + emailed to the updated contacts.
|
||||
- **Two-phase least-privilege migration** forced by GTIware co-residency; routed to two independent models for concurrence before treating it as execution-ready.
|
||||
- **Parked the migration** once recon showed the website is a cross-office + accounting + payroll + msdb hub on one sysadmin credential — a clean least-priv login barely exists; the durable fix is architectural, and a full network recon is the prerequisite.
|
||||
|
||||
## Problems Encountered
|
||||
|
||||
- `icacls /T` on 43,602 files exceeded the agent command timeout repeatedly → ran it detached (Start-Process) and polled a completion marker.
|
||||
- App-pool recycle didn't clear `401.3` → root cause was the cached IUSR token; `iisreset` resolved it.
|
||||
- `web.config` rollback didn't fix the outage (it was an ACL problem, not config) — preserved Tom's file rather than discarding his hardening.
|
||||
- Recon hit the wrong SQL instance first (default instance via `localhost -E`); the cards instance is the named `GTISQL` on `:3436`.
|
||||
- `SYSTEM` (agent) is **not** sysadmin on `:3436`, so the authoritative login/role/`tom` map there still needs the `tom` credential or a sysadmin Windows login.
|
||||
|
||||
## Configuration Changes
|
||||
|
||||
**Glaztech WWW (192.168.8.72) — production server, via GuruRMM:**
|
||||
- `IIS_IUSRS` local group: added `NT AUTHORITY\IUSR` and `IIS APPPOOL\glaztech_new` (were missing — non-default).
|
||||
- `icacls "D:\web\glaztech_4" /grant "IIS_IUSRS:(OI)(CI)(RX)" /T` — granted app-pool/anonymous read across the content tree (completes E1; `Everyone:(R)` remains removed).
|
||||
- `iisreset /restart`.
|
||||
- `Web.config` rolled back to `Web.config.bak-20260604-170500` (the working 6/3 version); Tom's 6/4 edit preserved as `Web.config.broken-20260605-084124` (NOT the cause; contains debug=false + security headers + secure cookies, for re-apply after coordinating with Tom).
|
||||
|
||||
**Repo:**
|
||||
- Created `clients/glaztech/reports/2026-06-05-least-privilege-db-migration-scope.md` (v0.3 — scope + Grok/Gemini review + recon findings; PARKED).
|
||||
- Created this session log.
|
||||
|
||||
**Syncro:**
|
||||
- #32378 comment 417493519 ("Website service restored", customer-visible) + resend 417494988 to updated contacts.
|
||||
|
||||
## Infrastructure & Servers (Glaztech SQL topology — corrected 2026-06-05)
|
||||
|
||||
- **GTI-INV-SQL** (machine, 192.168.8.62, at "INV - Involta" colo) runs **3 SQL instances**:
|
||||
- **Default instance** — SQL Server **2008 R2** (10.50.2550), DBs: `qqest`, ReportServer(+TempDB), system. `NT AUTHORITY\SYSTEM` IS sysadmin here.
|
||||
- **`GTI-INV-SQL\GTISQL` on port 3436** — SQL Server **2012** (11.0.7507) — **the cards/website instance**, ~57 DBs: `glaz_prod_<office>` + `_archive` + `_web` for **alb, boi, brl, corp, den, elp, phx, shp, slc, tuc**, PDF stores (`glaz_pdf*`), `gti_samsara`, `qqest`. `xp_cmdshell=1`. `SYSTEM` is NOT sysadmin here.
|
||||
- **Third instance on 192.168.8.62,3430.**
|
||||
- **Linked servers (from `:3436`):** 192.168.0.54,55181 · 192.168.0.55,55181 (`mas_gti`/Sage) · 192.168.8.52,3436 (backup) · 192.168.8.212,3436 (backup) · 192.168.8.62,3430 · `GLAZ\TIMEFORCE` (`qqest`). All data-access enabled; default-instance linked logins map `(default-all) → tom`.
|
||||
- **Website (`WWW`) connection strings (active, all `user id=tom`):** glaz_prod (tuc), glaz_prod_phx/_slc/_elp/_den/_alb/_boi/_brl/_shp/_corp, glaz_pdf, glaz_pdf_corp — all on 192.168.8.62,3436; **`mas_gti` @ 192.168.0.55,55181** (Sage); **`qqest` via glaz\timeforce** (payroll); **`msdb` @ 192.168.8.62,3436** (`glaztech_jobs`).
|
||||
- **Old (commented-out) topology** (matches Mike's per-site memory): per-office ports `glaz,3430` (tuc) / 3432 (phx) / 3438 (slc) / 3431 (elp) / 3435 (den) / 3433 (alb) / 3437 (boi) / 3439 (brl); `sql1,3436`; `sql3,3430`. Since consolidated.
|
||||
- **`tom`** SQL login created 2017-12-30; sysadmin; cross-mesh remote login. Password in WWW `Web.config` (cleartext, NOT vaulted; redacted from all artifacts).
|
||||
- **GuruRMM Glaztech agents (4):** WWW (455a1bc7), GTI-INV-SQL (869e56b4), GTI-INV-DC, GTI-INV-DC1.
|
||||
|
||||
## Commands & Outputs (key)
|
||||
|
||||
```
|
||||
# Outage fix (WWW, via /rmm)
|
||||
net localgroup IIS_IUSRS IUSR /add ; net localgroup IIS_IUSRS "IIS APPPOOL\glaztech_new" /add
|
||||
icacls "D:\web\glaztech_4" /grant "IIS_IUSRS:(OI)(CI)(RX)" /T /C /Q # 43,602 files; run detached
|
||||
iisreset /restart
|
||||
# verify: last 60 IIS requests -> 55x200 0x401 0x500 ; external 200 on apex/www/login/images
|
||||
|
||||
# SQL recon (GTI-INV-SQL, via /rmm, sqlcmd -S localhost[,3436] -E, read-only)
|
||||
SELECT name FROM sys.databases # default inst: qqest/ReportServer/system ; :3436: ~57 glaz_prod_* etc
|
||||
SELECT name,data_source FROM sys.servers WHERE is_linked=1 # the mesh (0.54/0.55/8.52/8.212/8.62:3430/timeforce)
|
||||
# Web.config connectionStrings on WWW -> all user id=tom across all offices + mas_gti + qqest + msdb
|
||||
```
|
||||
|
||||
## Pending / Incomplete Tasks
|
||||
|
||||
- **Least-privilege `tom` migration — PARKED** pending a **full network recon** (Mike enrolling site servers in RMM) to map how the estate fits together; the website's true footprint (all offices + accounting + payroll + msdb) makes a clean least-priv login barely viable → reconsider in favor of the architectural fixes (assessment items 16-17, 22). Scope: `clients/glaztech/reports/2026-06-05-least-privilege-db-migration-scope.md` (v0.3). Coord todo `aebaf751`.
|
||||
- **Authoritative `:3436` login/role recon** still needs the `tom` credential (or a sysadmin Windows login) — `SYSTEM` isn't sysadmin there.
|
||||
- **Emergency containment (E2-E5)** still open and now more urgent: E2 (rotate `glaztech\administrator` + strip cleartext from `msdb` job steps) is a hard prerequisite because the website legitimately holds an `msdb` connection. E3 (disable `xp_cmdshell` — still =1 on `:3436`), E4 (de-priv SQL Agent / disable `sa`), E5 (firewall/RealVNC).
|
||||
- **Tom's `web.config` hardening** (debug=false + security headers + secure cookies) preserved on WWW; re-apply after coordinating with Tom.
|
||||
- Whole web-app remediation remains **parked on #32378 (Waiting on Customer)**.
|
||||
- Failed-login detection/lockout (H5): options laid out; waiting on Tom (app-side) — no evidence of active attack.
|
||||
|
||||
## Reference Information
|
||||
|
||||
- Scope doc: `clients/glaztech/reports/2026-06-05-least-privilege-db-migration-scope.md`
|
||||
- Assessment: `clients/glaztech/reports/2026-06-03-website-security-assessment.md`
|
||||
- Ticket: #32378 (id 112111185), Waiting on Customer. Comments 417493519 + 417494988.
|
||||
- Coord todos: `aebaf751` (least-priv `tom` migration), `6d15fc88` (E2-E4 containment).
|
||||
- GuruRMM: WWW agent `455a1bc7-1c29-42bc-b597-fa1e64f08eec`; GTI-INV-SQL agent `869e56b4-e8ed-4808-8c88-782d1577c152`.
|
||||
Reference in New Issue
Block a user