142 lines
17 KiB
Markdown
142 lines
17 KiB
Markdown
# Session Log — 2026-06-03 — Glaz-Tech Industries
|
|
|
|
## User
|
|
- **User:** Mike Swanson (mike)
|
|
- **Machine:** GURU-5070
|
|
- **Role:** admin
|
|
|
|
---
|
|
|
|
## Session Summary
|
|
|
|
Worked a series of Glaztech web/payment issues on server `WWW` (192.168.8.72 / public 65.113.52.88), all via the GuruRMM agent, ending in a full security assessment of the website and its databases.
|
|
|
|
First, diagnosed why `shannon@glaztech.com` was not receiving MailProtector quarantine digests (sent as `noreply@azcomputerguru.com`). A `Get-MessageTraceV2` (Exchange Operator app, tenant 82931e3c) over the last 10 days showed 629 digests delivered to ~60 recipients and **zero to Shannon**. Confirmed Shannon is a real, active mailbox receiving normal mail. Conclusion: not an Exchange/EOP problem (the morning SCL-bypass rule is irrelevant to her) — MailProtector is not generating a digest for her; the fix is on the MailProtector side (provision her / enable the Spam Summary). Vault has no MailProtector credentials, so it is a manual partner-portal task.
|
|
|
|
Handled an emergency apex 404: `glaztech.com` returned 404 (HTTP.sys) while `www.glaztech.com` returned 200, because IIS site `glaztech_new` had a host-header binding only for `www`. Added apex bindings (http:80 + https:443, reusing the existing SAN cert that already covers both names). Then added a URL Rewrite HTTP→HTTPS 301 redirect with a `/.well-known/acme-challenge/` exclusion so Let's Encrypt (Certify The Web, HTTP-01) renewal is unaffected. Created and invoiced ticket #32376 (1h remote, prepaid).
|
|
|
|
Diagnosed and fixed the CyberSource credit-card payment outage ("Could not create SSL/TLS secure channel," 20 attempts / 0 success across all offices). Root cause: CyberSource (PNC processor) disabled TLS 1.0/1.1; the .NET payment library on Server 2019 negotiated old TLS. Applied `SchUseStrongCrypto=1` + `SystemDefaultTlsVersions=1` to both `.NETFramework\v4.0.30319` hives and recycled the `glaztech_new` app pool. Verified via the payments DB that card payments resumed (a real credit-card approval at 09:36 after the fix; eCheck path was never affected). Created and invoiced ticket #32377 (1.5h emergency remote, prepaid).
|
|
|
|
Performed a deep security assessment (read-only; no card numbers or passwords retrieved). Findings are severe: full PANs and CVV stored in plaintext, ~9,000+ plaintext passwords, SQL injection via a fake `quo()` escaper in payment pages, reflected XSS, debug/error disclosure, the production server doubling as a dev workstation, remote-access sprawl (incl. RealVNC 4.2.8), and a listener accepting TLS 1.0/1.1. Produced two reports and a high-priority client ticket (#32378) with a plain-language, point-by-point public writeup for Steve/Tom. Also traced why cards are stored (card-on-file invoice auto-pay via `gt_auto_process_2020.dll`) and confirmed the exposure is contained to the web app's databases (not Sage 100). Billed 1h remote to #32378 and left it open for remediation.
|
|
|
|
---
|
|
|
|
## Key Decisions
|
|
|
|
- **MailProtector digest = MailProtector-side issue.** Message trace proved Shannon gets no digest at all; the EOP SCL-bypass rule (created earlier) only helps users whose digests were being spam-filtered. No Exchange change made for Shannon.
|
|
- **Apex fix = add binding, not redirect.** Restored apex by adding host-header bindings to the existing site (apex serves same content as www) — fastest restore; cert already SAN-covered apex so no cert change.
|
|
- **HTTP→HTTPS redirect with ACME exclusion.** Used IIS URL Rewrite (installed) with a `/.well-known/acme-challenge/` negate condition so Certify/Let's Encrypt HTTP-01 renewal is not broken. 301 permanent.
|
|
- **Payment fix applied server-side first.** Set the .NET TLS keys + app-pool recycle (the standard fix). Verified success via the payments DB rather than waiting on Tom; the app-code `ServicePointManager.SecurityProtocol = Tls12` fallback was staged but not needed (DB confirmed live card approvals post-fix).
|
|
- **Security findings classified, never exfiltrated.** All card/password determinations used aggregate `CASE`/`COUNT` only — no PAN or password value was selected or displayed.
|
|
- **#32378 left OPEN (In Progress)** at Mike's direction for likely remediation work; billed but not resolved/closed.
|
|
- **Tokenization is the card remediation** — preserves the auto-pay feature while removing all stored PAN/CVV.
|
|
|
|
---
|
|
|
|
## Problems Encountered
|
|
|
|
- **`Get-MessageTraceV2` not exposed via `adminapi/InvokeCommand` initially** — investigator-exo token 401'd; the exchange-op token works and the cmdlet runs via InvokeCommand with a ≤10-day window constraint. The ExchangeOnlineManagement PowerShell module is NOT installed on GURU-5070 (used the REST InvokeCommand path instead).
|
|
- **PowerShell quoting/parse errors** in several RMM probes (escaped quotes inside `$(...)`, `foreach | Format-Table` empty-pipe, DBNull comparison). Resolved by collecting into variables, moving counts out of inline subexpressions, and DBNull-safe formatters.
|
|
- **Local apex test 404'd** because the binding is IP-scoped to 192.168.8.72 and `localhost` is 127.0.0.1 — verified externally instead (301/200/acme-404 all correct).
|
|
- **Syncro long-comment heredoc broke bash** (apostrophes inside `$(cat <<EOF)`). Resolved by writing the body to a file with a direct heredoc redirect, collapsing newlines to `<br>`, and posting via `jq -n --rawfile`.
|
|
|
|
---
|
|
|
|
## Configuration Changes
|
|
|
|
**On server WWW (192.168.8.72) — via GuruRMM:**
|
|
- IIS site `glaztech_new`: added bindings `http/192.168.8.72:80:glaztech.com` and `https/192.168.8.72:443:glaztech.com` (apex; reuses cert thumbprint `2684b127f039be5cb5adea6f3f616c111824e4dc`).
|
|
- `D:\web\glaztech_4\web.config`: inserted URL Rewrite rule "HTTP to HTTPS" (301, host-preserving, with `/.well-known/acme-challenge/` exclusion) as the first rule. Backup at `D:\web\glaztech_4\web.config.bak-20260603-090701`.
|
|
- Registry: `SchUseStrongCrypto=1` + `SystemDefaultTlsVersions=1` (DWORD) added to BOTH `HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319` and `HKLM\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v4.0.30319`; recycled app pool `glaztech_new`.
|
|
|
|
**Repo (ClaudeTools):**
|
|
- Created `clients/glaztech/reports/2026-06-03-pci-cardholder-data-finding.md`
|
|
- Created `clients/glaztech/reports/2026-06-03-website-security-assessment.md` (incl. attack-path + card data-flow sections)
|
|
|
|
**Syncro tickets:** #32376 (apex 404 + redirect), #32377 (payment TLS), #32378 (security assessment — left open).
|
|
|
|
---
|
|
|
|
## Credentials & Secrets
|
|
|
|
- **MailProtector / CloudFilter partner portal** — NO credentials in the vault. ACG manages it via manual partner-portal login. (Action: consider vaulting for future automation.)
|
|
- **MSP360 API** (unrelated, used 6/2): `msp-tools/msp360-api.sops.yaml`.
|
|
- **Exchange Operator app** (message trace): vault `msp-tools/computerguru-exchange-operator.sops.yaml`, app `b43e7342-5b4b-492f-890f-bb5a4f7f40e9`, tenant `82931e3c-de7a-4f74-87f7-fe714be1f160`; token via `get-token.sh <tenant> exchange-op`.
|
|
- **Glaztech SQL** (observed in site Web.config, NOT vaulted): server `192.168.8.62,3436`, login `tom` (password in `D:\web\glaztech_4\Web.config` connectionStrings), 15 per-office databases (`glaz_prod`, `glaz_prod_phx`, …, `glaz_prod_corp`) + `mas_gti` (Sage 100) + `qqest` (TimeForce).
|
|
|
|
---
|
|
|
|
## Infrastructure & Servers
|
|
|
|
- **WWW** — Windows Server 2019 Standard, build 10.0.17763.8755 (patched through May 2026). Local IP 192.168.8.72, public 65.113.52.88. IIS 10.0, .NET 4.8. Site `glaztech_new` → `D:\web\glaztech_4` (full VB.NET source on disk, not precompiled). LE cert `CN=www.glaztech.com`, SAN glaztech.com + www, expires 2026-08-19, via Certify The Web (HTTP-01). GuruRMM agent id `455a1bc7-1c29-42bc-b597-fa1e64f08eec`.
|
|
- **SQL** — `192.168.8.62,3436`, 15 office DBs + Sage 100 (`mas_gti`) + TimeForce (`qqest`).
|
|
- **Payment gateway** — CyberSource REST (`api.cybersource.com/pts/v2/payments`) as PNC merchant processor (card path); CyberSource SOAP toolkit for eCheck/ACH (`cybs.log`).
|
|
- **Email/spam** — MailProtector inbound filter `glaztech-com.inbound.emailservice.io`; digests from `noreply@azcomputerguru.com`.
|
|
|
|
---
|
|
|
|
## Commands & Outputs
|
|
|
|
- Apex 404: `curl -I https://glaztech.com` → 404 `Microsoft-HTTPAPI/2.0`; post-fix → 200; `/.well-known/acme-challenge/probe` over http → 404 plain (renewal safe).
|
|
- Payment TLS test (WWW → api.cybersource.com): TLS1.0/1.1 FAIL (SSPI), **TLS1.2 OK**; post-fix DB shows credit-card approval id 84455 at 09:36 (`web_payment_header`, glaz_prod_corp).
|
|
- Auth proc `get_web_accesslevel`: `where acct_no=@username and web_password=@passwd` (plaintext password compare).
|
|
- `quo()`: `Return "'" + stext + "'"` (no escaping → SQLi).
|
|
- `get_cc_data`: `select * from cc_file where acct_no=@acctno` (returns full PAN+CVV; IDOR-shaped).
|
|
- Card storage classification: `cc_file` ~780 rows total (plaintext PAN, 0 encrypted); `cc_file.cc_code` 50/54 CVV in Tucson; `cof_payments_header` Phoenix 14,496 rows / 11,794 plaintext PAN; `web_security` ~9,000+ plaintext passwords (corp 6,017 + tuc 3,012), 0 hash-like.
|
|
- Message trace (Shannon): 629 noreply digests in 10 days to ~60 recipients, 0 to shannon@glaztech.com.
|
|
|
|
---
|
|
|
|
## Pending / Incomplete Tasks
|
|
|
|
- **MailProtector:** provision/enable Shannon's Spam Summary in the MailProtector partner portal (manual). The earlier `gtimail@glaztech.com` daily digest failure (Priority-1 reject rule) is still open with Steve.
|
|
- **#32378 remediation (open ticket):** quick wins ready on Mike's go — purge stored CVV (`cc_file.cc_code`, backup-first, needs explicit sign-off), set `debug="false"` + `customErrors="On"`, remove RealVNC 4.2.8 + stale ScreenConnect, disable TLS 1.0/1.1. Then: hash passwords + stop emailing them, parameterize payment-page SQL (kill `quo()`), tokenize card-on-file (CyberSource vault), separate dev from prod, least-privilege DB.
|
|
- **Payment fix:** monitor that card successes continue; Tom code fallback (`SecurityProtocol = Tls12`) staged if needed.
|
|
- **Backups contain plaintext cards** — cleanup must address DB backups, not just live data.
|
|
|
|
---
|
|
|
|
## Reference Information
|
|
|
|
- Tickets: #32376 (id 112107224, apex 404+redirect, Resolved, 1h remote), #32377 (id 112109440, CyberSource TLS, Resolved, 1.5h emergency remote), #32378 (id 112111185, security assessment, **In Progress**, 1h remote).
|
|
- Glaztech Syncro customer id 143932; **prepaid block 26.5 → 22.25 hrs** today.
|
|
- Reports: `clients/glaztech/reports/2026-06-03-pci-cardholder-data-finding.md`, `…-website-security-assessment.md`.
|
|
- GuruRMM client id `d857708c-5713-4ee5-a314-679f86d2f9f9`; WWW agent `455a1bc7-1c29-42bc-b597-fa1e64f08eec`.
|
|
- TLS fix pattern (legacy .NET → modern gateway): `SchUseStrongCrypto=1` + `SystemDefaultTlsVersions=1` both `.NETFramework\v4.0.30319` hives + app-pool recycle.
|
|
- LE-safe HTTP→HTTPS redirect: URL Rewrite rule with `{REQUEST_URI}` `^/\.well-known/acme-challenge/` negate condition.
|
|
|
|
---
|
|
|
|
## Update: 15:52 PT — Payrilla/CyberSource reconciliation + Sage finding
|
|
|
|
Mike reported Glaztech "no longer uses CyberSource, switched to Payrilla (secure card storage)." Verified the live system rather than take it at face value — and the technical reality contradicts the belief:
|
|
|
|
- **Website is STILL on CyberSource/PNC.** No `payrilla`/`payrix`/`paya`/`payroc` references anywhere in `D:\web\glaztech_4` code or config. Live payment pages remain `online-payment-pnc.aspx` / `quick-pay-pnc.aspx` → `api.cybersource.com`. `web_payment_header` shows live `CC-WebPayment-PNC` Approved/Error transactions **today** (2026-06-03 14:47 Approved, 15:07 Error) + E-Check.
|
|
- **Plaintext cards STILL being written daily.** `cc_file` last write: Tucson **2026-06-03 14:15** (8/60d), Phoenix **2026-06-03 10:19** (14/60d). Ongoing accumulation, not legacy.
|
|
- **Sage 100 (`mas_gti`) CC module is DISABLED** — `SY_Company` (Company GTI) `CreditCardEnable=N`, `CreditCardVaultOnlyID` empty, `AR_CustomerCreditCard` = **0 rows**. Corrects an earlier in-session assumption that Sage actively tokenizes via Paya: the tokenization columns exist in the Sage schema but are **unused**. Sage stores no cardholder data at all (strengthens the containment finding).
|
|
- **Payrilla/Paya is not visible in either system we can reach.** If in use, it's a separate channel (standalone virtual terminal / different portal / manual) that does NOT cover the website. This is the likely source of the client's "we moved to Payrilla for everything" belief — true at most for some standalone tool, NOT for the website (the system storing plaintext cards).
|
|
- **Anomaly:** `cc_file` now returns "Invalid object name" in the `corp` DB though it existed there this morning (3 rows) — needs a second look; per-office DBs (tuc/phx) unaffected and still writing.
|
|
|
|
**Actions:** updated `clients/glaztech/reports/2026-06-03-website-security-assessment.md` (added "Current State Verified — 2026-06-03" section; corrected two Sage lines from "uses tokenization / materially compliant" to "CC module disabled, 0 stored cards"). Posted public+emailed client comment on **#32378** (id 417040624) reflecting the live, ongoing risk and that Payrilla is not yet implemented on the site. Drafted a further client clarification (the Sage/Payrilla visibility gap) — **pending Mike's go** before posting (would be the 4th #32378 email today).
|
|
|
|
**Open:** confirm with Payrilla which flows actually route through them; investigate the corp `cc_file` anomaly; remediation path reframed to "migrate the website's card-on-file to the chosen processor's token vault, stop writing `cc_file`/`cof_payments_header`, purge plaintext + backups, decommission CyberSource."
|
|
|
|
---
|
|
|
|
## Update: 19:32 PT — Tom (dev) reply thread: storage = GTIware, website = sysadmin access vector (TOP CRITICAL)
|
|
|
|
Tom (Glaztech's internal IT / **author of the GTIware/GlazGTI PSA**) replied to #32378 asking where the website stores data, stating he stores nothing from the website's online payment system. Investigated and **Tom is correct** — and it surfaced the single worst finding of the engagement.
|
|
|
|
- **The website stores NO cards.** Zero `cc_file`/`save_cc_data` references in the site's `.aspx`/`.vb`; the quick-pay pages even carry a disclaimer "Glaz-Tech Industries does not save nor record any card info." The site uses the shared `gt_utilities` namespace (209 refs) for general plumbing only.
|
|
- **The cards are written by GTIware** (Tom's PSA): the `cc_file`/`save_cc_data` logic is in compiled libraries `glaztech_utilities_2020.dll` + `gt_auto_process_2020.dll` (deployed in the site's `Bin` but not called by the site). Recent `cc_file` rows are stamped with **staff** usernames (`Victoria`, `Bryce`, `Diana`) and notes like "RUN CARD WHEN REQUESTED" — a back-office save-a-card workflow.
|
|
- **TOP CRITICAL (now C0 in the report): the website connects to the GTIware SQL server as a `sysadmin`.** The site's `Web.config` uses SQL auth `user id=tom` (login id 267, a **named SQL login, member of the `sysadmin` role — NOT the built-in `sa`**, created 2018; password embedded in `Web.config`). The SQL host is **`GTI-INV-SQL` (192.168.8.62,3436)**, shared by the website and GTIware, with **46 databases** (all offices' `glaz_prod*` + `*_archive` + PDF stores + `qqest` TimeForce + `gti_samsara` + system DBs). Because SQLi runs as the connecting login, the website's `quo()` SQL-injection executes as sysadmin. **Cross-DB confirmed live:** from the Tucson connection, read other offices' `glaz_prod_phx.dbo.cc_file` (141), `glaz_prod_den` (190), `glaz_prod_elp` (179). So one guessed website login + the SQLi = full read/write/destroy of the entire GTIware instance + `xp_cmdshell` OS takeover of the SQL server.
|
|
- `get_cc_data` is `SELECT * FROM cc_file WHERE acct_no=@acctno` → returns full PAN+CVV; IDOR-shaped on `@acctno`. Other sysadmin logins on the instance: `GTI-INV-SQL\Administrator`, `NT SERVICE\*`, `sa` (enabled), `tom`.
|
|
|
|
**Actions:**
|
|
- Added **C0** (top critical) to the website security assessment with the full attack tree (steal/destroy/manipulate/OS-takeover) and required remediation (least-privilege web login that cannot see GTIware data; separate/partition the website DB from GTIware; stop storing cards — call processor API directly or tokenize; never store CVV (PCI 3.2); encrypt at rest; fix SQLi + lockout).
|
|
- **Corrected both reports' attribution**: storage = GTIware internal PSA (staff-operated), website = access path; Sage CC module disabled (0 cards), not a CHD location.
|
|
- Posted **plain-English reply to Tom** on #32378 (public+emailed, comment 417070212): leads with "how we found it" (a website-only review reached internal GTIware card data → that reachability IS the flaw), explains the sysadmin+SQLi chain plainly, lists the four fixes.
|
|
- #32378 remains **Waiting on Customer**.
|
|
|
|
**Open / next:** Tom to remediate (pull sysadmin login from the site → least-privilege; isolate website DB from GTIware; tokenize / stop storing; never store CVV; fix SQLi + lockout). The corp `cc_file` "Invalid object name" anomaly still unexplained. ACG can execute the quick wins (CVV purge, least-privilege login, debug-off) on request.
|