claudetools/clients/glaztech/session-logs/2026-06-04-session.md

# Glaztech — Session Log 2026-06-04

## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin

## Session Summary

Performed a read-only intrusion / brute-force log review on the Glaztech web server (`WWW`, agent `455a1bc7-1c29-42bc-b597-fa1e64f08eec`, Windows Server 2019, agent v0.6.54) via GuruRMM, following up on the 2026-06-03 website security assessment. Question: is there evidence anyone tried to brute-force the website logins or the server itself.

Analyzed 7 days of IIS logs (`C:\inetpub\logs\LogFiles\W3SVC4`, ~52,000 requests, May 29 – Jun 4). Identified the two login endpoints (`/customer_login.aspx`, `/emp/employee-login.aspx`), then broke POSTs down per source IP with HTTP status codes. An initial pass flagged the employee portal as suspicious (381 "failures"/HTTP 200 vs 77 "successes"/302 — an inverted ratio vs the customer login). Pulling full per-IP request timelines corrected this: the employee login returns **HTTP 200 on both success and failure** (no redirect-on-success), so status code alone does not indicate a failed staff login. The top "suspect" IP `160.3.157.9` proved to be a single legitimate employee on an iPhone checking timecards. No brute-force or credential-stuffing signature exists on either endpoint.

Also reviewed the Windows Security event log (4625 failed logons / 4624 successes, 7-day window): only 13 failed logons, all LogonType 3 (SMB) from internal LAN IPs, zero external, no RDP failures, and no successful remote logons from public IPs — indicating RDP/SMB are not internet-exposed and nothing got in at the OS auth layer.

Folded the findings into the security record: added **Appendix A — Intrusion / Brute-Force Log Review (2026-06-04)** to `clients/glaztech/reports/2026-06-03-website-security-assessment.md`, and extended finding **H5** with a detection-blind-spot note (200-on-both + no lockout + no failed-login logging = slow guessing would be invisible).

## Key Decisions

- **Used IIS request logs as the primary evidence source, not the Windows Security log**, because the app's custom session-based auth never generates Windows logon events. The Security log was used only to rule out RDP/SMB brute force against the host.
- **Did not treat HTTP 200 on the employee login as a failure** once the timelines showed 200-on-success. Reported the corrected interpretation rather than the alarming-but-wrong intermediate read.
- **Flagged the HTTP 500 bursts** as worth an app-side look (possible SQLi error path per C3) but explicitly classified them as not-a-brute-force, to avoid overstating the threat.
- **Recorded the detection gap as the actionable outcome** — the review found no attacker, but confirmed that a slow guessing attack would currently be undetectable; reinforces the existing #32378 remediation (add lockout + failed-login logging).

## Problems Encountered

- **`jq -n --arg` heredoc capture glitched** when dispatching the Security-log script (parse error / empty command). Resolved by writing the PowerShell to `.claude/tmp/seclog.ps1` and passing it with `jq --rawfile`, which is robust against embedded quoting.
- **IIS logs do not record POST bodies**, so attempted usernames/passwords are not recoverable — noted as a limitation; confirming which accounts (if any) were targeted would require app-level auth logging that does not exist.

## Configuration Changes

- Modified: `clients/glaztech/reports/2026-06-03-website-security-assessment.md` — added Appendix A (2026-06-04 log review); extended H5 with detection-blind-spot note + failed-login-logging fix; updated Status line.
- Created: `clients/glaztech/session-logs/2026-06-04-session.md` (this file).
- Local-only (gitignored): `.claude/current-mode` set to `infra`; transient `.claude/tmp/seclog.ps1`, `.claude/tmp/rmm.token`, `.claude/tmp/rmm.cmd`.

## Credentials & Secrets

- None discovered or created. RMM admin credential read from vault `infrastructure/gururmm-server.sops.yaml` (unchanged).

## Infrastructure & Servers

- **WWW** (Glaztech web server) — internal `192.168.8.72`, public `65.113.52.88`, Windows Server 2019 (build 17763), GuruRMM agent `455a1bc7-1c29-42bc-b597-fa1e64f08eec` (v0.6.54), client "Glaztech Industries" / site "TUS - Tucson".
- Active IIS site: `W3SVC4` (log dir `C:\inetpub\logs\LogFiles\W3SVC4`; daily `u_exYYMMDD.log`). Older sites W3SVC1/2/3 are stale (last writes 2018–2022).
- SQL backend (context only, not touched): `192.168.8.62,3436` (`GTI-INV-SQL`).

## Commands & Outputs

- IIS analysis dispatched as PowerShell via GuruRMM (`POST /api/agents/<id>/command`), parsing `#Fields`-keyed log lines for `cs-method`, `cs-uri-stem`, `c-ip`, `sc-status`, `date`, `time`, `cs(User-Agent)`.
- Key counts: customer login 2,547×302 / 78×200 / 5×500; employee login 77×302 / 381×200 / 6×500; 740 distinct IPs hit the login endpoints; 241 distinct IPs hit the employee login.
- Security log: span 2026-03-31 .. 2026-06-04; 4625 in last 7d = 13, all type 3, all internal, usernames = 12 blank + 1 `tomabens`; 4624 external type 3/8/10 = none.

## Pending / Incomplete Tasks

- **App-side investigation of the HTTP 500 bursts** on post-login pages (possible SQLi error path) — IPs `201.146.179.166`, `64.178.182.162`, `205.185.107.49`, `172.87.137.60`. Not yet handed to Tom.
- **#32378 (Waiting on Customer)** unchanged: add account lockout + failed-login logging (now explicitly tied to this review), fix SQLi, least-privilege the `tom`/`sysadmin` DB login, separate website DB from GTIware, stop storing cards / never CVV, encrypt at rest.
- Optional confirming check offered but not run: enumerate listening/forwarded ports on `WWW` to turn "RDP almost certainly not exposed" into "confirmed."
- Carried over: `corp` DB `cc_file` "Invalid object name" anomaly; confirm Payrilla payment-flow scope.

## Reference Information

- Report: `clients/glaztech/reports/2026-06-03-website-security-assessment.md` (Appendix A)
- Companion: `clients/glaztech/reports/2026-06-03-pci-cardholder-data-finding.md`
- Wiki: `wiki/clients/glaztech.md`
- Ticket: Syncro **#32378** (Glaztech website security — Waiting on Customer)
- RMM API: `http://172.16.3.30:3001` | agent id `455a1bc7-1c29-42bc-b597-fa1e64f08eec`

---

## Update: 09:38 PT — Deep host + SQL-infrastructure recon (Grok gap-analysis loop)

### Session Summary
Ran a collaborative gap-analysis loop (Claude + Grok CLI, 4 Grok turns on session `019e9351-…`) plus 3 read-only GuruRMM pulls on the `WWW` host to find what the website security assessment had not yet checked. Grok identified overlooked surfaces; Claude pulled the requested read-only host telemetry + SQL **metadata**; results were fed back to Grok each turn. Findings were folded into the assessment report. All read-only; no CHD and no secret values intentionally retrieved.

### Key Findings (new vs. prior report)
- **GTIware card engine is co-resident in the public IIS worker on the `tom` sysadmin connection.** `gt_auto_process_2020.dll` / `glaztech_utilities_2020.dll` live in `D:\web\glaztech_4\Bin\`; their `.dll.config` carry no own connection strings → they use the website's `tom` connections. `w3wp` PID 13752 = `IIS APPPOOL\glaztech_new`.
- **`xp_cmdshell = 1` (enabled) on `GTI-INV-SQL`** → instant OS RCE from a `tom` SQLi (`clr`/`Ole`/`Ad Hoc`/`remote admin` all 0).
- **SQL Agent service runs as `Administrator@glaztech.com`** (domain-admin class).
- **Cleartext `glaztech\administrator` domain-admin password embedded in `msdb` SQL Agent backup-copy job steps** (`xp_cmdshell 'net use … /user:glaztech\administrator <pw>'`) — readable by any sysadmin/`tom` connection. **Value redacted from all artifacts (sed-scrubbed from the recon output); flagged for rotation.**
- **7 linked servers** (remote-login + data-access) across `192.168.0.x` and `192.168.8.x` (`mas_gti`, `qqest`/TimeForce, backup hosts, `GTI-INV-SQL\GTISQL`).
- **`sa` login enabled; all 47 DBs `is_encrypted=False` (no TDE);** CHD `.bak` backups copied to `\\192.168.8.52\sql_backup\`.
- **`Web.config` holds two processors' secrets in cleartext** (CyberSource `cybs.*` + on-disk signing keys; PNC `pnc_key/pin/token`); **`Everyone:(R)` ACL** on web root + `bin` + `Web.config`.
- Host firewall Domain=On, **Private/Public=Off**; **TLS 1.0 explicitly `Enabled=1`**; new endpoints `/webhooks` (Samsara receiver) + `/webhooks1`.
- Server-side billing engine `d:\sql_jobs\bin\gt_console_apps.exe` (`cp`/`is`/`oa`/`lo`) run by the Agent (domain-admin).
- Extra hosts surfaced: `\\192.168.0.147\web\glaztech_4`, backups `8.52`/`8.212`, MAS `0.55`, linked `0.54`.

### Corrections / retractions
- An initial pass mis-flagged the `kgc7jt` ScreenConnect Cloud instance as "foreign/unrecognized third-party access" (draft finding **C6**). Per Mike: ScreenConnect (both), Datto RMM/EDR, Syncro, Splashtop are **ACG's sanctioned management stack**. **C6 fully retracted.** RealVNC is **Steve's** tool, kept as **H2** strictly for its end-of-life age (RealVNC 4.x ≈2009, listening 5900/5800).
- `machineKey` is AutoGenerate (not hardcoded); the app-pool identity is least-privilege locally — the privilege issue is the SQL `tom` login, not the Windows identity.

### Key Decisions
- Folded findings into the existing report as a new **C0-Extended** section + findings **C5/H9/H10** + an **Emergency-containment** roadmap bucket (E1–E5) ahead of all other work, rather than a separate report.
- Domain-admin password returned incidentally in a job-step preview was scrubbed from the on-disk recon artifact and never written to the report (no-secrets constraint).

### Configuration Changes
- Edited `clients/glaztech/reports/2026-06-03-website-security-assessment.md` (347 → 415 lines): method/date, Exec summary, findings table (+C5/H9/H10, −C6), C0 attack chain, new **C0-Extended** section, H2 rewrite, "Why the Cards Are Stored" nuance, Assumptions, PCI mapping, Emergency-containment bucket + structural items 22–26, status footer.

### Commands & Outputs
- GuruRMM read-only PowerShell to `WWW` (agent `455a1bc7…`): **recon1** (IIS/app-pool/tasks/services/GTIware hunt/ACLs/net/firewall/TLS/certs), **recon2** (w3wp identity, GTIware `.dll.config` safe-parse, SQL metadata via the app's `tom` connection — `sys.configurations`/`dm_server_services`/`sys.servers`/`msdb` jobs+backups/principals/`sys.databases`), **recon3** (ScreenConnect probe).
- Grok CLI session `019e9351-ed1c-7bc3-b171-b4cf4b53745d` (4 turns) for gap analysis + corrected remediation sequencing.

### Pending / Incomplete Tasks
- Coord todos filed: **6d15fc88** (rotate `glaztech\administrator` + purge `msdb` cleartext use; disable `xp_cmdshell`/`sa`), **aebaf751** (least-privilege `tom` migration).
- Recon the second host `\\192.168.0.147\web\glaztech_4`; lock down `/webhooks`.
- Carried: `corp` DB `cc_file` anomaly; IDOR on `get_cc_data`; Payrilla scope.

### Reference Information
- Grok session `019e9351-ed1c-7bc3-b171-b4cf4b53745d`; SQL host `GTI-INV-SQL` `192.168.8.62,3436` (instance `GTISQL`).
- Coord todos `6d15fc88-db4f-4a35-a76a-a5a6a9f50795`, `aebaf751-d778-423f-a84b-314fbb294f30`.

---

## Update: 19:07 PT — Glaztech infra remediation blitz via RMM (dev-tool removal, web hardening, domain time fix, ACL, sa)

### Session Summary
Executed a large batch of Glaztech remediation through GuruRMM. WWW was already enrolled; mid-session Mike enrolled the **DCs + SQL server**, which unlocked the domain/SQL-side work. All actions were RMM-driven, verified, and caused **no outages**. Several changes were scheduled (off-hours) with backup + health-check + auto-rollback. One acute item (msdb plaintext domain-admin removal) is **paused awaiting method approval**.

### Completed (all verified, no outage)
- **Dev tooling removed from WWW (H1):** VS 2015 + 2022 (~15.6 GB reclaimed, via bootstrapper `/uninstall /quiet /norestart`), IIS Express, Notepad++, OpenSSL, RealDownloader (+ scheduled task). Archived D7x (`D:\d7`, `D:\d7x Resources`), `D:\3rd Party Tools`, `D:\Scripts`, `D:\bin` (CyberSource SDK sample), and web-root `Old_code`/`Old_bin`/26 `.pdb` to **`D:\_removed_devtools_2026-06-04\`** (reversible). One-time reboot finalized the VS `PendingFileRenameOperations` (cleared).
- **WWW Web.config hardening** (scheduled task `ACG-WebConfigHarden-20260604` @ box-17:05, applied + survived the 17:15 reboot): `debug=false`; security headers `X-Content-Type-Options: nosniff`, `X-Frame-Options: SAMEORIGIN`, `Referrer-Policy: strict-origin-when-cross-origin`, `Strict-Transport-Security: max-age=31536000`; `httpCookies httpOnlyCookies=true requireSSL=true`; **CORS scoped to `<location path="emails">` (Origin:* Methods:GET)** and the site-wide wildcard CORS removed. Backup `D:\web\glaztech_4\Web.config.bak-20260604-170500`. Live headers verified.
- **Domain time fixed end-to-end:** PDC **GTI-INV-DC** was syncing from the Hyper-V host (VM IC provider) and drifting → re-pointed to **external NTP (pool.ntp.org)**, `VMICTimeProvider` disabled, marked reliable. **GTI-INV-DC1** → follows PDC. **WWW** (was `Local CMOS Clock`/free-running, ~8 min slow) re-registered (`w32tm /unregister`+`/register`) → PDC, clock **stepped +8 min**. **GTI-INV-SQL** → DC1. All four converged within ~3 s. Kerberos-skew resolved.
- **WWW `Everyone:(R)` ACL (E1):** removed from **`Web.config` + `bin`** (granted `IIS_IUSRS` + `IIS APPPOOL\glaztech_new` RX first; site stayed HTTP 301). Public static content (`emails/`,`images/`) left as a low-priority slower sweep.
- **GTI-INV-SQL: built-in `sa` disabled** (re-check showed **0 real user sessions**; the 29 "active" sessions were all `is_user_process=0` system sessions). Done via WWW's app `tom` connection (SYSTEM-on-SQL is not sysadmin).

### Key Findings
- **WWW clock** was never syncing (free-running) — ~6–8 min slow; surfaced when Mike noticed it. **PDC** itself was VM-host-timed, not NTP.
- **Forest = `glaztech.local` (root) + `glaztech.com` (child).** **NS4.glaztech.local holds the Schema-master FSMO but is a DEAD server** (per Mike) → orphaned FSMO; external NTP on GTI-INV-DC is correct (can't chain to dead root).
- **CORS:** the wildcard `Access-Control-Allow-Origin: *` was only used by cross-origin loads of `/emails/` assets (IIS logs: 188 OPTIONS, 181 → `/emails/`; none to the API/payment surface) → scoped to `/emails`.
- **msdb cleartext cred:** **11 TSQL backup-copy job steps** embed `net use \\192.168.8.52|.212\sql_backup\... /user:glaztech\administrator <pw> /persistent:yes` + a `copy`. They run as the SQL **engine** service account (machine acct, no share access) → can't just blank the creds. **0 existing SQL credentials/proxies; SQL Agent service account = `Administrator@glaztech.com` (domain admin).**

### Key Decisions
- **Web.config / ACL health checks must hit the real binding `http://192.168.8.72/` (Host: www.glaztech.com), NOT `127.0.0.1`** — the site binds to the LAN IP only. Caught + fixed the scheduled apply's health check at box-17:03, ~90 s before the 17:05 run (the 127.0.0.1 check would have false-rolled-back the change). **Reusable rule for future WWW scripts.**
- ACL fix scoped to `Web.config`+`bin` (the secrets/assemblies) instead of a slow full-tree `/T` (static content is public anyway).
- All scheduled/unattended changes built with backup + post-change health-check + auto-rollback; reachability-gated for the PDC NTP change (rollback to host time if NTP unreachable — it was reachable).

### PENDING — pick up next session
- **msdb plaintext removal — AWAITING GO on method.** Recommended: **SQL Credential + Agent CmdExec proxy** (encrypt the pw in `sys.credentials`, convert the 11 steps to CmdExec-under-proxy, drop inline creds; decoupled from Agent privilege; `ALTER CREDENTIAL` after rotation). Alt: `cmdkey` + strip inline. Test-first + snapshot originals to admin-only file (deleted after) + verify a copy works.
- **Rotate `glaztech\administrator`** — Mike coordinating with **Steve** (deferred). Identify all consumers first.
- Gated/heavier: **disable `xp_cmdshell`** (blocked until the 11 backup-copy steps are reworked — they depend on it); **disable TLS 1.0/1.1 on WWW** (needs reboot); **full web-root `Everyone` sweep** (low pri); **seize/clean Schema-master FSMO off dead NS4**; de-privilege the SQL Agent account.
- WWW one-time scheduled tasks `ACG-WebConfigHarden-20260604` + `ACG-Reboot-VSCleanup-20260604` both fired Result=0 (can be deleted or left).

### Infrastructure / Reference
- **Glaztech RMM agents** (client "Glaztech Industries"): `WWW` 455a1bc7 (site TUS-Tucson), `GTI-INV-DC` 0337e973 (**PDC**, INV-Involta), `GTI-INV-DC1` ffcaafac, `GTI-INV-SQL` 869e56b4. (NOT Glaztech: `SAGE-SQL`=Dataforth, `ACG-DC16`=ACG, `VWP-DC1`=VWP.)
- Domain `glaztech.com` (member servers); forest root `glaztech.local` (NS4, dead). Backup file servers `\\192.168.8.52\sql_backup`, `\\192.168.8.212\sql_backup`. SQL instance `GTISQL` @ `192.168.8.62,3436`.
- On-WWW logs: `C:\temp\{vs_uninstall, devtools_groupB, groupC, acl_fix, acl_fix2, sa_via_tom, webconfig_apply}.log`; on DCs/SQL: `C:\temp\timefix_*.log`.
- Local scripts (this machine): `C:\Users\guru\AppData\Local\Temp\grok_glaztech\*.ps1`.
- Coord locks held: `clients/glaztech:glaztech/domain-time` (61cd25f2), `clients/glaztech:WWW/devtools-removal` (c4226bac).

---

## Update: 19:28 PT — msdb plaintext-cred removal (Credential+Proxy) — PAUSED mid-flight (stopping for the night)

### Summary
Began removing the plaintext `glaztech\administrator` password from the **11 msdb backup-copy job steps** via the user-approved **SQL Credential + Agent CmdExec proxy** method. Phase 1 (create credential + proxy, test, snapshot, generate proposed commands) ran — but the **proxy authentication failed**. Decision on how to finish is deferred to next session. **The 11 production backup steps were NOT modified; backups run exactly as before.** User chose to leave the staged objects in place tonight (not cleaned up).

### EXACT STATE on GTI-INV-SQL (resume point)
- **11/11 backup-copy steps UNCHANGED** (still inline `net use ... /user:glaztech\administrator <pw>`); **0 converted**. Backups functional, untouched.
- **3 inert objects created in Phase 1 (NOT wired to any job, no effect on backups) — LEFT IN PLACE:**
  - Credential **`glaztech-backup-share`** (IDENTITY=`glaztech\administrator`, pw encrypted) — UNUSED.
  - Proxy **`ACG-BackupShareProxy`** (granted to CmdExec) — UNUSED.
  - Rollback table **`msdb.dbo.acg_jobstep_backup_20260604`** (11 rows = original step defs, **contains the plaintext pw**).
  - Temp job `ACG-ProxyTest` already deleted.
- **NEXT SESSION FIRST STEP:** decide method (below), then either reuse these or **DROP all 3** (`DROP CREDENTIAL [glaztech-backup-share]`; `sp_delete_proxy 'ACG-BackupShareProxy'`; `DROP TABLE msdb.dbo.acg_jobstep_backup_20260604`) to return to pristine. The snapshot table holds plaintext — drop it once we no longer need rollback.

### Key Findings
- **Embedded backup password is LIVE, not stale** — `net use \\192.168.8.52\sql_backup /user:glaztech\administrator <pw>` returned "command completed successfully"; share reachable. Current copies work.
- **Proxy auth fails with system error 1326 ("user name or password is incorrect")** despite the password being correct → a **logon-type mismatch**: `net use` = network logon (validated by the remote file server); a SQL Agent proxy does a **batch logon locally on GTI-INV-SQL**, resolving `glaztech\administrator` differently. NOT stale-pw, NOT a logon-right error (that'd be 1385).
- **The backup steps already "Executed as user: GLAZTECH\administrator"** — the **SQL Agent service account IS the domain admin**. So the simplest fix needs no stored credential at all (see decision).
- **Pre-existing bug (separate):** job **"Glaz Prod Differential (Hourly) to 8.62"** — **steps 13 & 14 FAIL** with `Incorrect syntax near 'EXEC [192.168.8.212,3436].msdb.dbo.sp_start_job N'` / `[192.168.8.52,3436]...` (broken linked-server job trigger). Backup-copy steps 11/12 succeed, but the job reports failure on 13/14. Worth investigating.

### DECISION PENDING (resume here) — how to remove the plaintext
- **RECOMMENDED: CmdExec-as-Agent.** Convert the 11 steps to CmdExec running the `copy` only (no `net use`, no creds). **Removes plaintext, stores NO password anywhere.** Works because the Agent = `GLAZTECH\administrator` (domain admin w/ share access). Generated commands already validated (copy cmds extract cleanly; **disabled step 1 → `rem` no-op**). Caveat: leans on the Agent's domain-admin privilege — but backups already depend on that today (zero regression); redo with a dedicated account + credential when the Agent is de-privileged.
- **ALT: troubleshoot the proxy** (try IDENTITY `administrator@glaztech.com` UPN, or batch-logon nuance) to keep the decoupled credential/proxy design.
- The generated new per-step commands (no secrets) are in the Phase-1 output / `C:\temp\msdb_cred_phase1.log` on WWW.

### Broader pending (Glaztech remediation backlog)
- **Rotate `glaztech\administrator`** — Mike coordinating with **Steve** (deferred). After rotation: `ALTER CREDENTIAL`/cmdkey or re-do CmdExec auth as needed.
- After plaintext removal → **disable `xp_cmdshell`** (was gated on these jobs).
- **Disable TLS 1.0/1.1 on WWW** (needs reboot); **full web-root `Everyone` sweep** (static, low pri); **seize/clean Schema-master FSMO off dead NS4**; **de-privilege the SQL Agent account**; investigate the hourly-job steps 13/14 bug.

### Reference
- Scripts (this machine): `C:\Users\guru\AppData\Local\Temp\grok_glaztech\{fix_msdb_phase1, diag_netuse, fix_www_acl2, sa_via_tom, fix_pdc_time, fix_dc1_time, fix_member_time, sched_webconfig, fix_apply}.ps1`.
- On GTI-INV-SQL (reach via WWW `tom` connection from `D:\web\glaztech_4\Web.config`, server `192.168.8.62`): cred `glaztech-backup-share`, proxy `ACG-BackupShareProxy`, table `msdb.dbo.acg_jobstep_backup_20260604`.