claudetools/clients/glaztech/session-logs/2026-06-04-session.md

Glaztech — Session Log 2026-06-04

User

• User: Mike Swanson (mike)
• Machine: GURU-5070
• Role: admin

Session Summary

Performed a read-only intrusion / brute-force log review on the Glaztech web server (WWW, agent 455a1bc7-1c29-42bc-b597-fa1e64f08eec, Windows Server 2019, agent v0.6.54) via GuruRMM, following up on the 2026-06-03 website security assessment. Question: is there evidence anyone tried to brute-force the website logins or the server itself.

Analyzed 7 days of IIS logs (C:\inetpub\logs\LogFiles\W3SVC4, ~52,000 requests, May 29 – Jun 4). Identified the two login endpoints (/customer_login.aspx, /emp/employee-login.aspx), then broke POSTs down per source IP with HTTP status codes. An initial pass flagged the employee portal as suspicious (381 "failures"/HTTP 200 vs 77 "successes"/302 — an inverted ratio vs the customer login). Pulling full per-IP request timelines corrected this: the employee login returns HTTP 200 on both success and failure (no redirect-on-success), so status code alone does not indicate a failed staff login. The top "suspect" IP 160.3.157.9 proved to be a single legitimate employee on an iPhone checking timecards. No brute-force or credential-stuffing signature exists on either endpoint.

Also reviewed the Windows Security event log (4625 failed logons / 4624 successes, 7-day window): only 13 failed logons, all LogonType 3 (SMB) from internal LAN IPs, zero external, no RDP failures, and no successful remote logons from public IPs — indicating RDP/SMB are not internet-exposed and nothing got in at the OS auth layer.

Folded the findings into the security record: added Appendix A — Intrusion / Brute-Force Log Review (2026-06-04) to clients/glaztech/reports/2026-06-03-website-security-assessment.md, and extended finding H5 with a detection-blind-spot note (200-on-both + no lockout + no failed-login logging = slow guessing would be invisible).

Key Decisions

• Used IIS request logs as the primary evidence source, not the Windows Security log, because the app's custom session-based auth never generates Windows logon events. The Security log was used only to rule out RDP/SMB brute force against the host.
• Did not treat HTTP 200 on the employee login as a failure once the timelines showed 200-on-success. Reported the corrected interpretation rather than the alarming-but-wrong intermediate read.
• Flagged the HTTP 500 bursts as worth an app-side look (possible SQLi error path per C3) but explicitly classified them as not-a-brute-force, to avoid overstating the threat.
• Recorded the detection gap as the actionable outcome — the review found no attacker, but confirmed that a slow guessing attack would currently be undetectable; reinforces the existing #32378 remediation (add lockout + failed-login logging).

Problems Encountered

• jq -n --arg heredoc capture glitched when dispatching the Security-log script (parse error / empty command). Resolved by writing the PowerShell to .claude/tmp/seclog.ps1 and passing it with jq --rawfile, which is robust against embedded quoting.
• IIS logs do not record POST bodies, so attempted usernames/passwords are not recoverable — noted as a limitation; confirming which accounts (if any) were targeted would require app-level auth logging that does not exist.

Configuration Changes

• Modified: clients/glaztech/reports/2026-06-03-website-security-assessment.md — added Appendix A (2026-06-04 log review); extended H5 with detection-blind-spot note + failed-login-logging fix; updated Status line.
• Created: clients/glaztech/session-logs/2026-06-04-session.md (this file).
• Local-only (gitignored): .claude/current-mode set to infra; transient .claude/tmp/seclog.ps1, .claude/tmp/rmm.token, .claude/tmp/rmm.cmd.

Credentials & Secrets

• None discovered or created. RMM admin credential read from vault infrastructure/gururmm-server.sops.yaml (unchanged).

Infrastructure & Servers

• WWW (Glaztech web server) — internal 192.168.8.72, public 65.113.52.88, Windows Server 2019 (build 17763), GuruRMM agent 455a1bc7-1c29-42bc-b597-fa1e64f08eec (v0.6.54), client "Glaztech Industries" / site "TUS - Tucson".
• Active IIS site: W3SVC4 (log dir C:\inetpub\logs\LogFiles\W3SVC4; daily u_exYYMMDD.log). Older sites W3SVC1/2/3 are stale (last writes 2018–2022).
• SQL backend (context only, not touched): 192.168.8.62,3436 (GTI-INV-SQL).

Commands & Outputs

• IIS analysis dispatched as PowerShell via GuruRMM (POST /api/agents/<id>/command), parsing #Fields-keyed log lines for cs-method, cs-uri-stem, c-ip, sc-status, date, time, cs(User-Agent).
• Key counts: customer login 2,547×302 / 78×200 / 5×500; employee login 77×302 / 381×200 / 6×500; 740 distinct IPs hit the login endpoints; 241 distinct IPs hit the employee login.
• Security log: span 2026-03-31 .. 2026-06-04; 4625 in last 7d = 13, all type 3, all internal, usernames = 12 blank + 1 tomabens; 4624 external type 3/8/10 = none.

Pending / Incomplete Tasks

• App-side investigation of the HTTP 500 bursts on post-login pages (possible SQLi error path) — IPs 201.146.179.166, 64.178.182.162, 205.185.107.49, 172.87.137.60. Not yet handed to Tom.
• #32378 (Waiting on Customer) unchanged: add account lockout + failed-login logging (now explicitly tied to this review), fix SQLi, least-privilege the tom/sysadmin DB login, separate website DB from GTIware, stop storing cards / never CVV, encrypt at rest.
• Optional confirming check offered but not run: enumerate listening/forwarded ports on WWW to turn "RDP almost certainly not exposed" into "confirmed."
• Carried over: corp DB cc_file "Invalid object name" anomaly; confirm Payrilla payment-flow scope.

Reference Information

• Report: clients/glaztech/reports/2026-06-03-website-security-assessment.md (Appendix A)
• Companion: clients/glaztech/reports/2026-06-03-pci-cardholder-data-finding.md
• Wiki: wiki/clients/glaztech.md
• Ticket: Syncro #32378 (Glaztech website security — Waiting on Customer)
• RMM API: http://172.16.3.30:3001 | agent id 455a1bc7-1c29-42bc-b597-fa1e64f08eec

---

Update: 09:38 PT — Deep host + SQL-infrastructure recon (Grok gap-analysis loop)

Session Summary

Ran a collaborative gap-analysis loop (Claude + Grok CLI, 4 Grok turns on session 019e9351-…) plus 3 read-only GuruRMM pulls on the WWW host to find what the website security assessment had not yet checked. Grok identified overlooked surfaces; Claude pulled the requested read-only host telemetry + SQL metadata; results were fed back to Grok each turn. Findings were folded into the assessment report. All read-only; no CHD and no secret values intentionally retrieved.

Key Findings (new vs. prior report)

• GTIware card engine is co-resident in the public IIS worker on the tom sysadmin connection. gt_auto_process_2020.dll / glaztech_utilities_2020.dll live in D:\web\glaztech_4\Bin\; their .dll.config carry no own connection strings → they use the website's tom connections. w3wp PID 13752 = IIS APPPOOL\glaztech_new.
• xp_cmdshell = 1 (enabled) on GTI-INV-SQL → instant OS RCE from a tom SQLi (clr/Ole/Ad Hoc/remote admin all 0).
• SQL Agent service runs as Administrator@glaztech.com (domain-admin class).
• Cleartext glaztech\administrator domain-admin password embedded in msdb SQL Agent backup-copy job steps (xp_cmdshell 'net use … /user:glaztech\administrator <pw>') — readable by any sysadmin/tom connection. Value redacted from all artifacts (sed-scrubbed from the recon output); flagged for rotation.
• 7 linked servers (remote-login + data-access) across 192.168.0.x and 192.168.8.x (mas_gti, qqest/TimeForce, backup hosts, GTI-INV-SQL\GTISQL).
• sa login enabled; all 47 DBs is_encrypted=False (no TDE); CHD .bak backups copied to \\192.168.8.52\sql_backup\.
• Web.config holds two processors' secrets in cleartext (CyberSource cybs.* + on-disk signing keys; PNC pnc_key/pin/token); Everyone:(R) ACL on web root + bin + Web.config.
• Host firewall Domain=On, Private/Public=Off; TLS 1.0 explicitly Enabled=1; new endpoints /webhooks (Samsara receiver) + /webhooks1.
• Server-side billing engine d:\sql_jobs\bin\gt_console_apps.exe (cp/is/oa/lo) run by the Agent (domain-admin).
• Extra hosts surfaced: \\192.168.0.147\web\glaztech_4, backups 8.52/8.212, MAS 0.55, linked 0.54.

Corrections / retractions

• An initial pass mis-flagged the kgc7jt ScreenConnect Cloud instance as "foreign/unrecognized third-party access" (draft finding C6). Per Mike: ScreenConnect (both), Datto RMM/EDR, Syncro, Splashtop are ACG's sanctioned management stack. C6 fully retracted. RealVNC is Steve's tool, kept as H2 strictly for its end-of-life age (RealVNC 4.x ≈2009, listening 5900/5800).
• machineKey is AutoGenerate (not hardcoded); the app-pool identity is least-privilege locally — the privilege issue is the SQL tom login, not the Windows identity.

Key Decisions

• Folded findings into the existing report as a new C0-Extended section + findings C5/H9/H10 + an Emergency-containment roadmap bucket (E1–E5) ahead of all other work, rather than a separate report.
• Domain-admin password returned incidentally in a job-step preview was scrubbed from the on-disk recon artifact and never written to the report (no-secrets constraint).

Configuration Changes

• Edited clients/glaztech/reports/2026-06-03-website-security-assessment.md (347 → 415 lines): method/date, Exec summary, findings table (+C5/H9/H10, −C6), C0 attack chain, new C0-Extended section, H2 rewrite, "Why the Cards Are Stored" nuance, Assumptions, PCI mapping, Emergency-containment bucket + structural items 22–26, status footer.

Commands & Outputs

• GuruRMM read-only PowerShell to WWW (agent 455a1bc7…): recon1 (IIS/app-pool/tasks/services/GTIware hunt/ACLs/net/firewall/TLS/certs), recon2 (w3wp identity, GTIware .dll.config safe-parse, SQL metadata via the app's tom connection — sys.configurations/dm_server_services/sys.servers/msdb jobs+backups/principals/sys.databases), recon3 (ScreenConnect probe).
• Grok CLI session 019e9351-ed1c-7bc3-b171-b4cf4b53745d (4 turns) for gap analysis + corrected remediation sequencing.

Pending / Incomplete Tasks

• Coord todos filed: 6d15fc88 (rotate glaztech\administrator + purge msdb cleartext use; disable xp_cmdshell/sa), aebaf751 (least-privilege tom migration).
• Recon the second host \\192.168.0.147\web\glaztech_4; lock down /webhooks.
• Carried: corp DB cc_file anomaly; IDOR on get_cc_data; Payrilla scope.

Reference Information

• Grok session 019e9351-ed1c-7bc3-b171-b4cf4b53745d; SQL host GTI-INV-SQL 192.168.8.62,3436 (instance GTISQL).
• Coord todos 6d15fc88-db4f-4a35-a76a-a5a6a9f50795, aebaf751-d778-423f-a84b-314fbb294f30.

---

Update: 19:07 PT — Glaztech infra remediation blitz via RMM (dev-tool removal, web hardening, domain time fix, ACL, sa)

Session Summary

Executed a large batch of Glaztech remediation through GuruRMM. WWW was already enrolled; mid-session Mike enrolled the DCs + SQL server, which unlocked the domain/SQL-side work. All actions were RMM-driven, verified, and caused no outages. Several changes were scheduled (off-hours) with backup + health-check + auto-rollback. One acute item (msdb plaintext domain-admin removal) is paused awaiting method approval.

Completed (all verified, no outage)

• Dev tooling removed from WWW (H1): VS 2015 + 2022 (~15.6 GB reclaimed, via bootstrapper /uninstall /quiet /norestart), IIS Express, Notepad++, OpenSSL, RealDownloader (+ scheduled task). Archived D7x (D:\d7, D:\d7x Resources), D:\3rd Party Tools, D:\Scripts, D:\bin (CyberSource SDK sample), and web-root Old_code/Old_bin/26 .pdb to D:\_removed_devtools_2026-06-04\ (reversible). One-time reboot finalized the VS PendingFileRenameOperations (cleared).
• WWW Web.config hardening (scheduled task ACG-WebConfigHarden-20260604 @ box-17:05, applied + survived the 17:15 reboot): debug=false; security headers X-Content-Type-Options: nosniff, X-Frame-Options: SAMEORIGIN, Referrer-Policy: strict-origin-when-cross-origin, Strict-Transport-Security: max-age=31536000; httpCookies httpOnlyCookies=true requireSSL=true; CORS scoped to <location path="emails"> (Origin: Methods:GET)* and the site-wide wildcard CORS removed. Backup D:\web\glaztech_4\Web.config.bak-20260604-170500. Live headers verified.
• Domain time fixed end-to-end: PDC GTI-INV-DC was syncing from the Hyper-V host (VM IC provider) and drifting → re-pointed to external NTP (pool.ntp.org), VMICTimeProvider disabled, marked reliable. GTI-INV-DC1 → follows PDC. WWW (was Local CMOS Clock/free-running, ~8 min slow) re-registered (w32tm /unregister+/register) → PDC, clock stepped +8 min. GTI-INV-SQL → DC1. All four converged within ~3 s. Kerberos-skew resolved.
• WWW Everyone:(R) ACL (E1): removed from Web.config + bin (granted IIS_IUSRS + IIS APPPOOL\glaztech_new RX first; site stayed HTTP 301). Public static content (emails/,images/) left as a low-priority slower sweep.
• GTI-INV-SQL: built-in sa disabled (re-check showed 0 real user sessions; the 29 "active" sessions were all is_user_process=0 system sessions). Done via WWW's app tom connection (SYSTEM-on-SQL is not sysadmin).

Key Findings

• WWW clock was never syncing (free-running) — ~6–8 min slow; surfaced when Mike noticed it. PDC itself was VM-host-timed, not NTP.
• Forest = glaztech.local (root) + glaztech.com (child). NS4.glaztech.local holds the Schema-master FSMO but is a DEAD server (per Mike) → orphaned FSMO; external NTP on GTI-INV-DC is correct (can't chain to dead root).
• CORS: the wildcard Access-Control-Allow-Origin: * was only used by cross-origin loads of /emails/ assets (IIS logs: 188 OPTIONS, 181 → /emails/; none to the API/payment surface) → scoped to /emails.
• msdb cleartext cred: 11 TSQL backup-copy job steps embed net use \\192.168.8.52|.212\sql_backup\... /user:glaztech\administrator <pw> /persistent:yes + a copy. They run as the SQL engine service account (machine acct, no share access) → can't just blank the creds. 0 existing SQL credentials/proxies; SQL Agent service account = Administrator@glaztech.com (domain admin).

Key Decisions

• Web.config / ACL health checks must hit the real binding http://192.168.8.72/ (Host: www.glaztech.com), NOT 127.0.0.1 — the site binds to the LAN IP only. Caught + fixed the scheduled apply's health check at box-17:03, ~90 s before the 17:05 run (the 127.0.0.1 check would have false-rolled-back the change). Reusable rule for future WWW scripts.
• ACL fix scoped to Web.config+bin (the secrets/assemblies) instead of a slow full-tree /T (static content is public anyway).
• All scheduled/unattended changes built with backup + post-change health-check + auto-rollback; reachability-gated for the PDC NTP change (rollback to host time if NTP unreachable — it was reachable).

PENDING — pick up next session

• msdb plaintext removal — AWAITING GO on method. Recommended: SQL Credential + Agent CmdExec proxy (encrypt the pw in sys.credentials, convert the 11 steps to CmdExec-under-proxy, drop inline creds; decoupled from Agent privilege; ALTER CREDENTIAL after rotation). Alt: cmdkey + strip inline. Test-first + snapshot originals to admin-only file (deleted after) + verify a copy works.
• Rotate glaztech\administrator — Mike coordinating with Steve (deferred). Identify all consumers first.
• Gated/heavier: disable xp_cmdshell (blocked until the 11 backup-copy steps are reworked — they depend on it); disable TLS 1.0/1.1 on WWW (needs reboot); full web-root Everyone sweep (low pri); seize/clean Schema-master FSMO off dead NS4; de-privilege the SQL Agent account.
• WWW one-time scheduled tasks ACG-WebConfigHarden-20260604 + ACG-Reboot-VSCleanup-20260604 both fired Result=0 (can be deleted or left).

Infrastructure / Reference

• Glaztech RMM agents (client "Glaztech Industries"): WWW 455a1bc7 (site TUS-Tucson), GTI-INV-DC 0337e973 (PDC, INV-Involta), GTI-INV-DC1 ffcaafac, GTI-INV-SQL 869e56b4. (NOT Glaztech: SAGE-SQL=Dataforth, ACG-DC16=ACG, VWP-DC1=VWP.)
• Domain glaztech.com (member servers); forest root glaztech.local (NS4, dead). Backup file servers \\192.168.8.52\sql_backup, \\192.168.8.212\sql_backup. SQL instance GTISQL @ 192.168.8.62,3436.
• On-WWW logs: C:\temp\{vs_uninstall, devtools_groupB, groupC, acl_fix, acl_fix2, sa_via_tom, webconfig_apply}.log; on DCs/SQL: C:\temp\timefix_*.log.
• Local scripts (this machine): C:\Users\guru\AppData\Local\Temp\grok_glaztech\*.ps1.
• Coord locks held: clients/glaztech:glaztech/domain-time (61cd25f2), clients/glaztech:WWW/devtools-removal (c4226bac).

---

Update: 19:28 PT — msdb plaintext-cred removal (Credential+Proxy) — PAUSED mid-flight (stopping for the night)

Summary

Began removing the plaintext glaztech\administrator password from the 11 msdb backup-copy job steps via the user-approved SQL Credential + Agent CmdExec proxy method. Phase 1 (create credential + proxy, test, snapshot, generate proposed commands) ran — but the proxy authentication failed. Decision on how to finish is deferred to next session. The 11 production backup steps were NOT modified; backups run exactly as before. User chose to leave the staged objects in place tonight (not cleaned up).

EXACT STATE on GTI-INV-SQL (resume point)

• 11/11 backup-copy steps UNCHANGED (still inline net use ... /user:glaztech\administrator <pw>); 0 converted. Backups functional, untouched.
• 3 inert objects created in Phase 1 (NOT wired to any job, no effect on backups) — LEFT IN PLACE:
  • Credential glaztech-backup-share (IDENTITY=glaztech\administrator, pw encrypted) — UNUSED.
  • Proxy ACG-BackupShareProxy (granted to CmdExec) — UNUSED.
  • Rollback table msdb.dbo.acg_jobstep_backup_20260604 (11 rows = original step defs, contains the plaintext pw).
  • Temp job ACG-ProxyTest already deleted.
• NEXT SESSION FIRST STEP: decide method (below), then either reuse these or DROP all 3 (DROP CREDENTIAL [glaztech-backup-share]; sp_delete_proxy 'ACG-BackupShareProxy'; DROP TABLE msdb.dbo.acg_jobstep_backup_20260604) to return to pristine. The snapshot table holds plaintext — drop it once we no longer need rollback.

Key Findings

• Embedded backup password is LIVE, not stale — net use \\192.168.8.52\sql_backup /user:glaztech\administrator <pw> returned "command completed successfully"; share reachable. Current copies work.
• Proxy auth fails with system error 1326 ("user name or password is incorrect") despite the password being correct → a logon-type mismatch: net use = network logon (validated by the remote file server); a SQL Agent proxy does a batch logon locally on GTI-INV-SQL, resolving glaztech\administrator differently. NOT stale-pw, NOT a logon-right error (that'd be 1385).
• The backup steps already "Executed as user: GLAZTECH\administrator" — the SQL Agent service account IS the domain admin. So the simplest fix needs no stored credential at all (see decision).
• Pre-existing bug (separate): job "Glaz Prod Differential (Hourly) to 8.62" — steps 13 & 14 FAIL with Incorrect syntax near 'EXEC [192.168.8.212,3436].msdb.dbo.sp_start_job N' / [192.168.8.52,3436]... (broken linked-server job trigger). Backup-copy steps 11/12 succeed, but the job reports failure on 13/14. Worth investigating.

DECISION PENDING (resume here) — how to remove the plaintext

• RECOMMENDED: CmdExec-as-Agent. Convert the 11 steps to CmdExec running the copy only (no net use, no creds). Removes plaintext, stores NO password anywhere. Works because the Agent = GLAZTECH\administrator (domain admin w/ share access). Generated commands already validated (copy cmds extract cleanly; disabled step 1 → rem no-op). Caveat: leans on the Agent's domain-admin privilege — but backups already depend on that today (zero regression); redo with a dedicated account + credential when the Agent is de-privileged.
• ALT: troubleshoot the proxy (try IDENTITY administrator@glaztech.com UPN, or batch-logon nuance) to keep the decoupled credential/proxy design.
• The generated new per-step commands (no secrets) are in the Phase-1 output / C:\temp\msdb_cred_phase1.log on WWW.

Broader pending (Glaztech remediation backlog)

• Rotate glaztech\administrator — Mike coordinating with Steve (deferred). After rotation: ALTER CREDENTIAL/cmdkey or re-do CmdExec auth as needed.
• After plaintext removal → disable xp_cmdshell (was gated on these jobs).
• Disable TLS 1.0/1.1 on WWW (needs reboot); full web-root Everyone sweep (static, low pri); seize/clean Schema-master FSMO off dead NS4; de-privilege the SQL Agent account; investigate the hourly-job steps 13/14 bug.

Reference

• Scripts (this machine): C:\Users\guru\AppData\Local\Temp\grok_glaztech\{fix_msdb_phase1, diag_netuse, fix_www_acl2, sa_via_tom, fix_pdc_time, fix_dc1_time, fix_member_time, sched_webconfig, fix_apply}.ps1.
• On GTI-INV-SQL (reach via WWW tom connection from D:\web\glaztech_4\Web.config, server 192.168.8.62): cred glaztech-backup-share, proxy ACG-BackupShareProxy, table msdb.dbo.acg_jobstep_backup_20260604.