claudetools/clients/glaztech/session-logs/2026-06-05-session.md

Glaztech Session Log — 2026-06-05

User

• User: Mike Swanson (mike)
• Machine: GURU-5070
• Role: admin

Session Summary

Two threads on Glaztech today: (1) a customer-facing website outage that was diagnosed and resolved, and (2) scoping the least-privilege SQL login migration (#32378 / coord todo aebaf751), which turned into a significant infrastructure-recon effort that materially changed the understanding of the environment.

Outage: glaztech.com (WWW, 192.168.8.72 / 65.113.52.88) returned site-wide HTTP 500 then 401. Root cause: a 2026-06-04 change applied the E1 hardening (removed the insecure Everyone:(R) from the web root D:\web\glaztech_4) but did not restore read access for the IIS serving accounts — and this server's IIS_IUSRS group was non-default, missing IUSR (the anonymous-auth identity). The IIS worker could load the app (DLLs + Web.config had explicit grants) but could not read the ~43,600-file content tree → 500.50 then 401.3 (access-denied-by-ACL). A web.config rollback was a red herring (preserved as Web.config.broken-20260605-084124). Fix (least-privilege; Everyone stays removed — completing E1 correctly): added IUSR + IIS APPPOOL\glaztech_new to IIS_IUSRS, granted IIS_IUSRS ReadAndExecute across the content tree (detached icacls — slow on 43.6K files, exceeded the agent command timeout), then iisreset to regenerate the cached IUSR token (an app-pool recycle was insufficient). Verified: last 60 requests 55x 200 / 0x 401 / 0x 500; external 200s on apex, www, customer_login.aspx, and product images. Posted a customer-visible #32378 comment ("Website service restored"), then resent it after Mike updated the ticket contacts (Tom primary, Alex + Steve CC).

Least-privilege scope: drafted a full scope for replacing the website's sysadmin login tom with a least-privilege login. The central constraint is GTIware co-residency (in-process card-engine DLLs share the website's Web.config connection and need cc_file), forcing a two-phase plan (Phase 1 strip sysadmin + the OS-RCE/domain-admin/cross-office/linked-server blast radius, keep cc_file; Phase 2 DENY cc_file after GTIware is decoupled). Routed the scope to Grok 4.3 and Gemini 3 Pro for independent adversarial review — both CONCUR with the two-phase approach and the matrix, with no material disagreement. Corrections folded in: don't DENY tempdb; audit linked-login catch-all mappings; dynamic SQL breaks ownership chaining; app-pool recycle to flush the ADO.NET pool on swap and rollback; alphanumeric-only interim password; fixed a rotation-gating contradiction; don't overstate Phase 1 (containment, not card-safety).

Recon — the picture changed: Mike suspected per-site SQL servers across a meshed 192.168.0-9.x network. Read-only RMM recon (GuruRMM agents on WWW + GTI-INV-SQL) showed it's centralized now (his memory matches the OLD topology, preserved in commented-out Web.config strings). All office data lives on one instance GTI-INV-SQL\GTISQL (192.168.8.62,3436, SQL 2012) — ~57 per-office DBs; a separate default instance (2008 R2) holds payroll/reporting; a third instance is on :3430. The website's Web.config revealed its true footprint: ~15 connection strings, all as tom, reaching all 10 offices' glaz_prod_* DBs + Sage accounting (mas_gti @ 192.168.0.55) + payroll (qqest) + msdb. That contradicts the scope's "DENY other offices/accounting/payroll/msdb" — the app legitimately uses them, shrinking Phase 1's value and favoring the architectural fixes. Mike parked the migration pending a real network recon (he's enrolling the site servers in RMM) and saved everything.

Key Decisions

• Outage fix completed E1 correctly (least-privilege ACL, Everyone stays removed) rather than reverting Tom's hardening — turned the incident into a remediation step done right.
• iisreset over app-pool recycle — the anonymous IUSR token is cached at the IIS service level; only a full reset picks up the new IIS_IUSRS membership.
• Detached icacls to escape the agent command timeout on the 43.6K-file ACL propagation.
• No blame in the ticket comment (Mike's instruction) — described only what was done; customer-visible + emailed to the updated contacts.
• Two-phase least-privilege migration forced by GTIware co-residency; routed to two independent models for concurrence before treating it as execution-ready.
• Parked the migration once recon showed the website is a cross-office + accounting + payroll + msdb hub on one sysadmin credential — a clean least-priv login barely exists; the durable fix is architectural, and a full network recon is the prerequisite.

Problems Encountered

• icacls /T on 43,602 files exceeded the agent command timeout repeatedly → ran it detached (Start-Process) and polled a completion marker.
• App-pool recycle didn't clear 401.3 → root cause was the cached IUSR token; iisreset resolved it.
• web.config rollback didn't fix the outage (it was an ACL problem, not config) — preserved Tom's file rather than discarding his hardening.
• Recon hit the wrong SQL instance first (default instance via localhost -E); the cards instance is the named GTISQL on :3436.
• SYSTEM (agent) is not sysadmin on :3436, so the authoritative login/role/tom map there still needs the tom credential or a sysadmin Windows login.

Configuration Changes

Glaztech WWW (192.168.8.72) — production server, via GuruRMM:

• IIS_IUSRS local group: added NT AUTHORITY\IUSR and IIS APPPOOL\glaztech_new (were missing — non-default).
• icacls "D:\web\glaztech_4" /grant "IIS_IUSRS:(OI)(CI)(RX)" /T — granted app-pool/anonymous read across the content tree (completes E1; Everyone:(R) remains removed).
• iisreset /restart.
• Web.config rolled back to Web.config.bak-20260604-170500 (the working 6/3 version); Tom's 6/4 edit preserved as Web.config.broken-20260605-084124 (NOT the cause; contains debug=false + security headers + secure cookies, for re-apply after coordinating with Tom).

Repo:

• Created clients/glaztech/reports/2026-06-05-least-privilege-db-migration-scope.md (v0.3 — scope + Grok/Gemini review + recon findings; PARKED).
• Created this session log.

Syncro:

• #32378 comment 417493519 ("Website service restored", customer-visible) + resend 417494988 to updated contacts.

Infrastructure & Servers (Glaztech SQL topology — corrected 2026-06-05)

• GTI-INV-SQL (machine, 192.168.8.62, at "INV - Involta" colo) runs 3 SQL instances:
  • Default instance — SQL Server 2008 R2 (10.50.2550), DBs: qqest, ReportServer(+TempDB), system. NT AUTHORITY\SYSTEM IS sysadmin here.
  • GTI-INV-SQL\GTISQL on port 3436 — SQL Server 2012 (11.0.7507) — the cards/website instance, ~57 DBs: glaz_prod_<office> + _archive + _web for alb, boi, brl, corp, den, elp, phx, shp, slc, tuc, PDF stores (glaz_pdf*), gti_samsara, qqest. xp_cmdshell=1. SYSTEM is NOT sysadmin here.
  • Third instance on 192.168.8.62,3430.
• Linked servers (from :3436): 192.168.0.54,55181 · 192.168.0.55,55181 (mas_gti/Sage) · 192.168.8.52,3436 (backup) · 192.168.8.212,3436 (backup) · 192.168.8.62,3430 · GLAZ\TIMEFORCE (qqest). All data-access enabled; default-instance linked logins map (default-all) → tom.
• Website (WWW) connection strings (active, all user id=tom): glaz_prod (tuc), glaz_prod_phx/_slc/_elp/_den/_alb/_boi/_brl/_shp/_corp, glaz_pdf, glaz_pdf_corp — all on 192.168.8.62,3436; mas_gti @ 192.168.0.55,55181 (Sage); qqest via glaz\timeforce (payroll); msdb @ 192.168.8.62,3436 (glaztech_jobs).
• Old (commented-out) topology (matches Mike's per-site memory): per-office ports glaz,3430 (tuc) / 3432 (phx) / 3438 (slc) / 3431 (elp) / 3435 (den) / 3433 (alb) / 3437 (boi) / 3439 (brl); sql1,3436; sql3,3430. Since consolidated.
• tom SQL login created 2017-12-30; sysadmin; cross-mesh remote login. Password in WWW Web.config (cleartext, NOT vaulted; redacted from all artifacts).
• GuruRMM Glaztech agents (4): WWW (455a1bc7), GTI-INV-SQL (869e56b4), GTI-INV-DC, GTI-INV-DC1.

Commands & Outputs (key)

# Outage fix (WWW, via /rmm)
net localgroup IIS_IUSRS IUSR /add ; net localgroup IIS_IUSRS "IIS APPPOOL\glaztech_new" /add
icacls "D:\web\glaztech_4" /grant "IIS_IUSRS:(OI)(CI)(RX)" /T /C /Q   # 43,602 files; run detached
iisreset /restart
# verify: last 60 IIS requests -> 55x200 0x401 0x500 ; external 200 on apex/www/login/images

# SQL recon (GTI-INV-SQL, via /rmm, sqlcmd -S localhost[,3436] -E, read-only)
SELECT name FROM sys.databases                  # default inst: qqest/ReportServer/system ; :3436: ~57 glaz_prod_* etc
SELECT name,data_source FROM sys.servers WHERE is_linked=1   # the mesh (0.54/0.55/8.52/8.212/8.62:3430/timeforce)
# Web.config connectionStrings on WWW -> all user id=tom across all offices + mas_gti + qqest + msdb

Pending / Incomplete Tasks

• Least-privilege tom migration — PARKED pending a full network recon (Mike enrolling site servers in RMM) to map how the estate fits together; the website's true footprint (all offices + accounting + payroll + msdb) makes a clean least-priv login barely viable → reconsider in favor of the architectural fixes (assessment items 16-17, 22). Scope: clients/glaztech/reports/2026-06-05-least-privilege-db-migration-scope.md (v0.3). Coord todo aebaf751.
• Authoritative :3436 login/role recon still needs the tom credential (or a sysadmin Windows login) — SYSTEM isn't sysadmin there.
• Emergency containment (E2-E5) still open and now more urgent: E2 (rotate glaztech\administrator + strip cleartext from msdb job steps) is a hard prerequisite because the website legitimately holds an msdb connection. E3 (disable xp_cmdshell — still =1 on :3436), E4 (de-priv SQL Agent / disable sa), E5 (firewall/RealVNC).
• Tom's web.config hardening (debug=false + security headers + secure cookies) preserved on WWW; re-apply after coordinating with Tom.
• Whole web-app remediation remains parked on #32378 (Waiting on Customer).
• Failed-login detection/lockout (H5): options laid out; waiting on Tom (app-side) — no evidence of active attack.

Reference Information

• Scope doc: clients/glaztech/reports/2026-06-05-least-privilege-db-migration-scope.md
• Assessment: clients/glaztech/reports/2026-06-03-website-security-assessment.md
• Ticket: #32378 (id 112111185), Waiting on Customer. Comments 417493519 + 417494988.
• Coord todos: aebaf751 (least-priv tom migration), 6d15fc88 (E2-E4 containment).
• GuruRMM: WWW agent 455a1bc7-1c29-42bc-b597-fa1e64f08eec; GTI-INV-SQL agent 869e56b4-e8ed-4808-8c88-782d1577c152.

Update: 10:35 PT — :3436 backup-job recon + Tom's architectural reply

Summary

Completed the :3436 SQL Agent job-definition recon (via tom credential, read-only) and reconciled two reachability checks. Then Tom replied to the partnership message with a substantial list of work he's already doing — which materially shifts ACG's role.

Recon findings

• 192.168.0.55,55181 (mas_gti linked server) is LIVE = GTI-FINANCESVR, SQL Server 2019 (15.0.4322.2). The website's accounting connection is current, NOT vestigial. "Old MAS90 dead" refers to the retired app/box, not this data path.
• SAGE2025 enrolled in RMM (Glaztech / TUS-Tucson) — new payroll server. Distinct from GTI-FINANCESVR. (This is why we asked Tom how qqest/payroll is used rather than assuming.)
• Cleartext domain-admin password (glaztech\administrator) sits in ~10-12 backup-job copy steps across 6 jobs on :3436. Pattern (redacted): exec xp_cmdshell 'net use \192.168.8.52\sql_backup\... /user:glaztech\administrator <PW> /persistent:yes' then xp_cmdshell 'copy d:\sql_backup\...\*.* \192.168.8.52\... /y'. Jobs: Glaz PDF Differential (Daily) to 8.62; Glaz PDF Full (weekly) to 8.52 + to 8.62; Glaz Prod Archive Full Monthly; Glaz Prod Differential (Hourly) to 8.62; Glaz Prod Full (Daily) to 8.62. Most also have a .212 copy step. Copy EndofWeek Backups uses xp_cmdshell for a LOCAL copy only (no creds).
• The BACKUP DATABASE steps themselves are clean TSQL → local disk. Only the push-to-share step carries the cleartext credential. Fix = replace each copy step with a CmdExec robocopy under the service account's own share access (no net use), OR BACKUP TO DISK=UNC directly. That removes the cleartext password AND the last xp_cmdshell dependency → unblocks disabling xp_cmdshell.
• The bulk of :3436 jobs are GTIware automation (gt_console_apps.exe modes + d:\sql_jobs*.bat) plus del \192.168.0.147\web\glaztech_4\pdf_output\*.pdf (confirms 192.168.0.147 = 2nd web host). Any dedicated Agent service account (E4) must retain DB + d:\sql_jobs + \192.168.0.147 + 8.52/8.212 backup-share access.

Tom's reply (strategy shift)

Tom independently: encrypted cc_number (cc_number_encrypted) + CVV (cc_code_encrypted) + website login passwords; is building separate web-only databases on 0.55 (no cc_file), with a new low-priv web login replacing his personal login; converting inline SQL to stored procs; long-term moving all DB access into *.dll library layer. This is the architectural fix we scoped — ACG's role shifts from "carry the app work" to validate/align + own backend infra (the :3436 cleartext/xp_cmdshell/sa/ domain-admin rotation, WAF, network segmentation).

Caveats raised to Tom (drafted reply, ask/remind tone)

1. CVV must not be retained at all even encrypted (PCI 3.2) — drop the column. (The one must-fix.)
2. Confirm PAN decryption key isolation (key out of web login's reach).
3. Confirm passwords are salted one-way hash vs reversible encrypt; retire any plaintext-password email.
4. Confirm new web login on 0.55 is scoped off the co-resident accounting (mas_gti) data.
5. Confirm back-office billing engine still points at 8.62 cc_file (cutover safety). Offered: the quo() fix-list as a stored-proc conversion checklist; help defining the web login grants.

Files

• Committed (fdcf014): clients/glaztech/reports/2026-06-05-tom-message-draft.md (final), 2026-06-05-quo-sql-fix-list.md (80 quo() sites / 15 files).
• New: clients/glaztech/reports/2026-06-05-tom-reply-draft.md + Outlook draft opened.

Pending

• Await Tom's answers (CVV drop, key location, hash vs encrypt, web-login isolation, billing path).
• ACG-owned Tier A still ours: recreate :3436 backup copy steps clean (CmdExec robocopy / dedicated service account on 8.52/8.212) -> disable xp_cmdshell -> disable sa -> rotate glaztech\administrator; WAF + SQL network segmentation. Sequence: E4 service acct -> clean copy steps -> xp_cmdshell off -> domain-admin rotation.
• Reference 58KB job dump: tool-results/b30gcchnr.txt (this session's transcript dir).