Mike Swanson
1aa9fcecad
- 2026-06-05-tom-reply2-draft.md (SENT): web-DB rearchitecture ack, CVV-no-paper correction, key-backup/escrow guidance, least-priv sync-job note - 2026-06-05-tom-quo-checklist.txt: clean 80-site quo() list sent to Tom - session log: TimeForce 2005->2008->2016 payroll chain (load-bearing, preserve) - guru-rmm submodule pointer -> dashboard redesign doc set (local) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
174 lines
16 KiB
Markdown
174 lines
16 KiB
Markdown
# Glaztech Session Log — 2026-06-05
|
|
|
|
## User
|
|
- **User:** Mike Swanson (mike)
|
|
- **Machine:** GURU-5070
|
|
- **Role:** admin
|
|
|
|
## Session Summary
|
|
|
|
Two threads on Glaztech today: (1) a customer-facing website outage that was diagnosed and resolved, and (2) scoping the least-privilege SQL login migration (#32378 / coord todo `aebaf751`), which turned into a significant infrastructure-recon effort that materially changed the understanding of the environment.
|
|
|
|
**Outage:** `glaztech.com` (WWW, 192.168.8.72 / 65.113.52.88) returned site-wide HTTP 500 then 401. Root cause: a 2026-06-04 change applied the E1 hardening (removed the insecure `Everyone:(R)` from the web root `D:\web\glaztech_4`) but did not restore read access for the IIS serving accounts — and this server's `IIS_IUSRS` group was non-default, missing `IUSR` (the anonymous-auth identity). The IIS worker could load the app (DLLs + Web.config had explicit grants) but could not read the ~43,600-file content tree → `500.50` then `401.3` (access-denied-by-ACL). A web.config rollback was a red herring (preserved as `Web.config.broken-20260605-084124`). Fix (least-privilege; `Everyone` stays removed — completing E1 correctly): added `IUSR` + `IIS APPPOOL\glaztech_new` to `IIS_IUSRS`, granted `IIS_IUSRS` ReadAndExecute across the content tree (detached `icacls` — slow on 43.6K files, exceeded the agent command timeout), then `iisreset` to regenerate the cached IUSR token (an app-pool recycle was insufficient). Verified: last 60 requests 55x 200 / 0x 401 / 0x 500; external 200s on apex, www, customer_login.aspx, and product images. Posted a customer-visible #32378 comment ("Website service restored"), then resent it after Mike updated the ticket contacts (Tom primary, Alex + Steve CC).
|
|
|
|
**Least-privilege scope:** drafted a full scope for replacing the website's `sysadmin` login `tom` with a least-privilege login. The central constraint is GTIware co-residency (in-process card-engine DLLs share the website's `Web.config` connection and need `cc_file`), forcing a two-phase plan (Phase 1 strip sysadmin + the OS-RCE/domain-admin/cross-office/linked-server blast radius, keep `cc_file`; Phase 2 DENY `cc_file` after GTIware is decoupled). Routed the scope to **Grok 4.3 and Gemini 3 Pro** for independent adversarial review — both CONCUR with the two-phase approach and the matrix, with no material disagreement. Corrections folded in: don't DENY `tempdb`; audit linked-login catch-all mappings; dynamic SQL breaks ownership chaining; app-pool recycle to flush the ADO.NET pool on swap and rollback; alphanumeric-only interim password; fixed a rotation-gating contradiction; don't overstate Phase 1 (containment, not card-safety).
|
|
|
|
**Recon — the picture changed:** Mike suspected per-site SQL servers across a meshed 192.168.0-9.x network. Read-only RMM recon (GuruRMM agents on WWW + GTI-INV-SQL) showed it's **centralized now** (his memory matches the OLD topology, preserved in commented-out Web.config strings). All office data lives on one instance `GTI-INV-SQL\GTISQL` (192.168.8.62,**3436**, SQL 2012) — ~57 per-office DBs; a separate default instance (2008 R2) holds payroll/reporting; a third instance is on `:3430`. The website's `Web.config` revealed its **true footprint**: ~15 connection strings, all as `tom`, reaching **all 10 offices' `glaz_prod_*` DBs + Sage accounting (`mas_gti` @ 192.168.0.55) + payroll (`qqest`) + `msdb`**. That contradicts the scope's "DENY other offices/accounting/payroll/msdb" — the app legitimately uses them, shrinking Phase 1's value and favoring the architectural fixes. Mike parked the migration pending a real network recon (he's enrolling the site servers in RMM) and saved everything.
|
|
|
|
## Key Decisions
|
|
|
|
- **Outage fix completed E1 correctly** (least-privilege ACL, `Everyone` stays removed) rather than reverting Tom's hardening — turned the incident into a remediation step done right.
|
|
- **`iisreset` over app-pool recycle** — the anonymous `IUSR` token is cached at the IIS service level; only a full reset picks up the new `IIS_IUSRS` membership.
|
|
- **Detached `icacls`** to escape the agent command timeout on the 43.6K-file ACL propagation.
|
|
- **No blame in the ticket comment** (Mike's instruction) — described only what was done; customer-visible + emailed to the updated contacts.
|
|
- **Two-phase least-privilege migration** forced by GTIware co-residency; routed to two independent models for concurrence before treating it as execution-ready.
|
|
- **Parked the migration** once recon showed the website is a cross-office + accounting + payroll + msdb hub on one sysadmin credential — a clean least-priv login barely exists; the durable fix is architectural, and a full network recon is the prerequisite.
|
|
|
|
## Problems Encountered
|
|
|
|
- `icacls /T` on 43,602 files exceeded the agent command timeout repeatedly → ran it detached (Start-Process) and polled a completion marker.
|
|
- App-pool recycle didn't clear `401.3` → root cause was the cached IUSR token; `iisreset` resolved it.
|
|
- `web.config` rollback didn't fix the outage (it was an ACL problem, not config) — preserved Tom's file rather than discarding his hardening.
|
|
- Recon hit the wrong SQL instance first (default instance via `localhost -E`); the cards instance is the named `GTISQL` on `:3436`.
|
|
- `SYSTEM` (agent) is **not** sysadmin on `:3436`, so the authoritative login/role/`tom` map there still needs the `tom` credential or a sysadmin Windows login.
|
|
|
|
## Configuration Changes
|
|
|
|
**Glaztech WWW (192.168.8.72) — production server, via GuruRMM:**
|
|
- `IIS_IUSRS` local group: added `NT AUTHORITY\IUSR` and `IIS APPPOOL\glaztech_new` (were missing — non-default).
|
|
- `icacls "D:\web\glaztech_4" /grant "IIS_IUSRS:(OI)(CI)(RX)" /T` — granted app-pool/anonymous read across the content tree (completes E1; `Everyone:(R)` remains removed).
|
|
- `iisreset /restart`.
|
|
- `Web.config` rolled back to `Web.config.bak-20260604-170500` (the working 6/3 version); Tom's 6/4 edit preserved as `Web.config.broken-20260605-084124` (NOT the cause; contains debug=false + security headers + secure cookies, for re-apply after coordinating with Tom).
|
|
|
|
**Repo:**
|
|
- Created `clients/glaztech/reports/2026-06-05-least-privilege-db-migration-scope.md` (v0.3 — scope + Grok/Gemini review + recon findings; PARKED).
|
|
- Created this session log.
|
|
|
|
**Syncro:**
|
|
- #32378 comment 417493519 ("Website service restored", customer-visible) + resend 417494988 to updated contacts.
|
|
|
|
## Infrastructure & Servers (Glaztech SQL topology — corrected 2026-06-05)
|
|
|
|
- **GTI-INV-SQL** (machine, 192.168.8.62, at "INV - Involta" colo) runs **3 SQL instances**:
|
|
- **Default instance** — SQL Server **2008 R2** (10.50.2550), DBs: `qqest`, ReportServer(+TempDB), system. `NT AUTHORITY\SYSTEM` IS sysadmin here.
|
|
- **`GTI-INV-SQL\GTISQL` on port 3436** — SQL Server **2012** (11.0.7507) — **the cards/website instance**, ~57 DBs: `glaz_prod_<office>` + `_archive` + `_web` for **alb, boi, brl, corp, den, elp, phx, shp, slc, tuc**, PDF stores (`glaz_pdf*`), `gti_samsara`, `qqest`. `xp_cmdshell=1`. `SYSTEM` is NOT sysadmin here.
|
|
- **Third instance on 192.168.8.62,3430.**
|
|
- **Linked servers (from `:3436`):** 192.168.0.54,55181 · 192.168.0.55,55181 (`mas_gti`/Sage) · 192.168.8.52,3436 (backup) · 192.168.8.212,3436 (backup) · 192.168.8.62,3430 · `GLAZ\TIMEFORCE` (`qqest`). All data-access enabled; default-instance linked logins map `(default-all) → tom`.
|
|
- **Website (`WWW`) connection strings (active, all `user id=tom`):** glaz_prod (tuc), glaz_prod_phx/_slc/_elp/_den/_alb/_boi/_brl/_shp/_corp, glaz_pdf, glaz_pdf_corp — all on 192.168.8.62,3436; **`mas_gti` @ 192.168.0.55,55181** (Sage); **`qqest` via glaz\timeforce** (payroll); **`msdb` @ 192.168.8.62,3436** (`glaztech_jobs`).
|
|
- **Old (commented-out) topology** (matches Mike's per-site memory): per-office ports `glaz,3430` (tuc) / 3432 (phx) / 3438 (slc) / 3431 (elp) / 3435 (den) / 3433 (alb) / 3437 (boi) / 3439 (brl); `sql1,3436`; `sql3,3430`. Since consolidated.
|
|
- **`tom`** SQL login created 2017-12-30; sysadmin; cross-mesh remote login. Password in WWW `Web.config` (cleartext, NOT vaulted; redacted from all artifacts).
|
|
- **GuruRMM Glaztech agents (4):** WWW (455a1bc7), GTI-INV-SQL (869e56b4), GTI-INV-DC, GTI-INV-DC1.
|
|
|
|
## Commands & Outputs (key)
|
|
|
|
```
|
|
# Outage fix (WWW, via /rmm)
|
|
net localgroup IIS_IUSRS IUSR /add ; net localgroup IIS_IUSRS "IIS APPPOOL\glaztech_new" /add
|
|
icacls "D:\web\glaztech_4" /grant "IIS_IUSRS:(OI)(CI)(RX)" /T /C /Q # 43,602 files; run detached
|
|
iisreset /restart
|
|
# verify: last 60 IIS requests -> 55x200 0x401 0x500 ; external 200 on apex/www/login/images
|
|
|
|
# SQL recon (GTI-INV-SQL, via /rmm, sqlcmd -S localhost[,3436] -E, read-only)
|
|
SELECT name FROM sys.databases # default inst: qqest/ReportServer/system ; :3436: ~57 glaz_prod_* etc
|
|
SELECT name,data_source FROM sys.servers WHERE is_linked=1 # the mesh (0.54/0.55/8.52/8.212/8.62:3430/timeforce)
|
|
# Web.config connectionStrings on WWW -> all user id=tom across all offices + mas_gti + qqest + msdb
|
|
```
|
|
|
|
## Pending / Incomplete Tasks
|
|
|
|
- **Least-privilege `tom` migration — PARKED** pending a **full network recon** (Mike enrolling site servers in RMM) to map how the estate fits together; the website's true footprint (all offices + accounting + payroll + msdb) makes a clean least-priv login barely viable → reconsider in favor of the architectural fixes (assessment items 16-17, 22). Scope: `clients/glaztech/reports/2026-06-05-least-privilege-db-migration-scope.md` (v0.3). Coord todo `aebaf751`.
|
|
- **Authoritative `:3436` login/role recon** still needs the `tom` credential (or a sysadmin Windows login) — `SYSTEM` isn't sysadmin there.
|
|
- **Emergency containment (E2-E5)** still open and now more urgent: E2 (rotate `glaztech\administrator` + strip cleartext from `msdb` job steps) is a hard prerequisite because the website legitimately holds an `msdb` connection. E3 (disable `xp_cmdshell` — still =1 on `:3436`), E4 (de-priv SQL Agent / disable `sa`), E5 (firewall/RealVNC).
|
|
- **Tom's `web.config` hardening** (debug=false + security headers + secure cookies) preserved on WWW; re-apply after coordinating with Tom.
|
|
- Whole web-app remediation remains **parked on #32378 (Waiting on Customer)**.
|
|
- Failed-login detection/lockout (H5): options laid out; waiting on Tom (app-side) — no evidence of active attack.
|
|
|
|
## Reference Information
|
|
|
|
- Scope doc: `clients/glaztech/reports/2026-06-05-least-privilege-db-migration-scope.md`
|
|
- Assessment: `clients/glaztech/reports/2026-06-03-website-security-assessment.md`
|
|
- Ticket: #32378 (id 112111185), Waiting on Customer. Comments 417493519 + 417494988.
|
|
- Coord todos: `aebaf751` (least-priv `tom` migration), `6d15fc88` (E2-E4 containment).
|
|
- GuruRMM: WWW agent `455a1bc7-1c29-42bc-b597-fa1e64f08eec`; GTI-INV-SQL agent `869e56b4-e8ed-4808-8c88-782d1577c152`.
|
|
|
|
## Update: 10:35 PT — :3436 backup-job recon + Tom's architectural reply
|
|
|
|
### Summary
|
|
Completed the `:3436` SQL Agent job-definition recon (via tom credential, read-only) and
|
|
reconciled two reachability checks. Then Tom replied to the partnership message with a
|
|
substantial list of work he's already doing — which materially shifts ACG's role.
|
|
|
|
### Recon findings
|
|
- **`192.168.0.55,55181` (mas_gti linked server) is LIVE** = `GTI-FINANCESVR`, SQL Server 2019
|
|
(15.0.4322.2). The website's accounting connection is current, NOT vestigial. "Old MAS90 dead"
|
|
refers to the retired app/box, not this data path.
|
|
- **`SAGE2025` enrolled in RMM** (Glaztech / TUS-Tucson) — new payroll server. Distinct from
|
|
GTI-FINANCESVR. (This is why we asked Tom how qqest/payroll is used rather than assuming.)
|
|
- **Cleartext domain-admin password (`glaztech\administrator`) sits in ~10-12 backup-job copy
|
|
steps** across 6 jobs on `:3436`. Pattern (redacted): `exec xp_cmdshell 'net use
|
|
\192.168.8.52\sql_backup\... /user:glaztech\administrator <PW> /persistent:yes'` then
|
|
`xp_cmdshell 'copy d:\sql_backup\...\*.* \192.168.8.52\... /y'`. Jobs: Glaz PDF Differential
|
|
(Daily) to 8.62; Glaz PDF Full (weekly) to 8.52 + to 8.62; Glaz Prod Archive Full Monthly;
|
|
Glaz Prod Differential (Hourly) to 8.62; Glaz Prod Full (Daily) to 8.62. Most also have a
|
|
`.212` copy step. `Copy EndofWeek Backups` uses xp_cmdshell for a LOCAL copy only (no creds).
|
|
- The `BACKUP DATABASE` steps themselves are clean TSQL → local disk. Only the push-to-share step
|
|
carries the cleartext credential. Fix = replace each copy step with a CmdExec robocopy under the
|
|
service account's own share access (no net use), OR BACKUP TO DISK=UNC directly. That removes the
|
|
cleartext password AND the last xp_cmdshell dependency → unblocks disabling xp_cmdshell.
|
|
- The bulk of `:3436` jobs are GTIware automation (`gt_console_apps.exe` modes + d:\sql_jobs\*.bat)
|
|
plus `del \192.168.0.147\web\glaztech_4\pdf_output\*.pdf` (confirms 192.168.0.147 = 2nd web host).
|
|
Any dedicated Agent service account (E4) must retain DB + d:\sql_jobs + \192.168.0.147 +
|
|
8.52/8.212 backup-share access.
|
|
|
|
### Tom's reply (strategy shift)
|
|
Tom independently: encrypted cc_number (cc_number_encrypted) + CVV (cc_code_encrypted) + website
|
|
login passwords; is building separate web-only databases on 0.55 (no cc_file), with a new low-priv
|
|
`web` login replacing his personal login; converting inline SQL to stored procs; long-term moving
|
|
all DB access into *.dll library layer. This is the architectural fix we scoped — ACG's role shifts
|
|
from "carry the app work" to validate/align + own backend infra (the :3436 cleartext/xp_cmdshell/sa/
|
|
domain-admin rotation, WAF, network segmentation).
|
|
|
|
### Caveats raised to Tom (drafted reply, ask/remind tone)
|
|
1. CVV must not be retained at all even encrypted (PCI 3.2) — drop the column. (The one must-fix.)
|
|
2. Confirm PAN decryption key isolation (key out of web login's reach).
|
|
3. Confirm passwords are salted one-way hash vs reversible encrypt; retire any plaintext-password email.
|
|
4. Confirm new `web` login on 0.55 is scoped off the co-resident accounting (mas_gti) data.
|
|
5. Confirm back-office billing engine still points at 8.62 cc_file (cutover safety).
|
|
Offered: the quo() fix-list as a stored-proc conversion checklist; help defining the web login grants.
|
|
|
|
### Files
|
|
- Committed (fdcf014): clients/glaztech/reports/2026-06-05-tom-message-draft.md (final),
|
|
2026-06-05-quo-sql-fix-list.md (80 quo() sites / 15 files).
|
|
- New: clients/glaztech/reports/2026-06-05-tom-reply-draft.md + Outlook draft opened.
|
|
|
|
### Pending
|
|
- Await Tom's answers (CVV drop, key location, hash vs encrypt, web-login isolation, billing path).
|
|
- ACG-owned Tier A still ours: recreate :3436 backup copy steps clean (CmdExec robocopy / dedicated
|
|
service account on 8.52/8.212) -> disable xp_cmdshell -> disable sa -> rotate glaztech\administrator;
|
|
WAF + SQL network segmentation. Sequence: E4 service acct -> clean copy steps -> xp_cmdshell off ->
|
|
domain-admin rotation.
|
|
- Reference 58KB job dump: tool-results/b30gcchnr.txt (this session's transcript dir).
|
|
|
|
## Update — Tom's answer on the payroll/qqest cross-SQL-version chain (2026-06-05)
|
|
|
|
Resolves the Thread-1 question from the :3436 backup-job recon (was it vestigial?). It is NOT —
|
|
it's the **TimeForce** payroll bridge and it's load-bearing:
|
|
- **TimeForce** (hourly employee timestamps) runs on **SQL 2005**.
|
|
- Data path: **2005 -> 2008 -> 2016**. The 2008 hop is a required transition (can't go 2005 -> 2012+
|
|
directly); from 2008 it's pushed to the active **2016** instance that runs most processes. The
|
|
chain also doubles as an **active backup** of TimeForce.
|
|
- Can't run TimeForce directly on 2016: the program is too old (compatibility issues) AND they have
|
|
**no installation media** to reinstall it. So Tom built the multi-hop chain rather than risk
|
|
breaking hourly payroll.
|
|
- Tom: "deal with this after we get the website security all fixed up." Agreed — out of scope now.
|
|
|
|
**Implications for our work:**
|
|
- The cross-version sync jobs we saw on :3436 are this TimeForce chain — **preserve them**; do NOT
|
|
touch the 2005->2008->2016 flow during the E-bucket / backup-copy cleanup.
|
|
- Our backup-job fix (recreate the cleartext domain-admin `net use` COPY steps cleanly) is a
|
|
SEPARATE concern from the TimeForce payroll chain — keep them distinct.
|
|
|
|
**Future remediation item (PARKED, post-website, agreed with Tom):** SQL 2005 is long EOL/unpatched
|
|
and the app has no reinstall media — a real fragility + security liability. Candidate for a later
|
|
named project (modern time-clock or a supported re-platform), not now.
|