11 KiB
Session Log — 2026-06-03 — Glaz-Tech Industries
User
- User: Mike Swanson (mike)
- Machine: GURU-5070
- Role: admin
Session Summary
Worked a series of Glaztech web/payment issues on server WWW (192.168.8.72 / public 65.113.52.88), all via the GuruRMM agent, ending in a full security assessment of the website and its databases.
First, diagnosed why shannon@glaztech.com was not receiving MailProtector quarantine digests (sent as noreply@azcomputerguru.com). A Get-MessageTraceV2 (Exchange Operator app, tenant 82931e3c) over the last 10 days showed 629 digests delivered to ~60 recipients and zero to Shannon. Confirmed Shannon is a real, active mailbox receiving normal mail. Conclusion: not an Exchange/EOP problem (the morning SCL-bypass rule is irrelevant to her) — MailProtector is not generating a digest for her; the fix is on the MailProtector side (provision her / enable the Spam Summary). Vault has no MailProtector credentials, so it is a manual partner-portal task.
Handled an emergency apex 404: glaztech.com returned 404 (HTTP.sys) while www.glaztech.com returned 200, because IIS site glaztech_new had a host-header binding only for www. Added apex bindings (http:80 + https:443, reusing the existing SAN cert that already covers both names). Then added a URL Rewrite HTTP→HTTPS 301 redirect with a /.well-known/acme-challenge/ exclusion so Let's Encrypt (Certify The Web, HTTP-01) renewal is unaffected. Created and invoiced ticket #32376 (1h remote, prepaid).
Diagnosed and fixed the CyberSource credit-card payment outage ("Could not create SSL/TLS secure channel," 20 attempts / 0 success across all offices). Root cause: CyberSource (PNC processor) disabled TLS 1.0/1.1; the .NET payment library on Server 2019 negotiated old TLS. Applied SchUseStrongCrypto=1 + SystemDefaultTlsVersions=1 to both .NETFramework\v4.0.30319 hives and recycled the glaztech_new app pool. Verified via the payments DB that card payments resumed (a real credit-card approval at 09:36 after the fix; eCheck path was never affected). Created and invoiced ticket #32377 (1.5h emergency remote, prepaid).
Performed a deep security assessment (read-only; no card numbers or passwords retrieved). Findings are severe: full PANs and CVV stored in plaintext, ~9,000+ plaintext passwords, SQL injection via a fake quo() escaper in payment pages, reflected XSS, debug/error disclosure, the production server doubling as a dev workstation, remote-access sprawl (incl. RealVNC 4.2.8), and a listener accepting TLS 1.0/1.1. Produced two reports and a high-priority client ticket (#32378) with a plain-language, point-by-point public writeup for Steve/Tom. Also traced why cards are stored (card-on-file invoice auto-pay via gt_auto_process_2020.dll) and confirmed the exposure is contained to the web app's databases (not Sage 100). Billed 1h remote to #32378 and left it open for remediation.
Key Decisions
- MailProtector digest = MailProtector-side issue. Message trace proved Shannon gets no digest at all; the EOP SCL-bypass rule (created earlier) only helps users whose digests were being spam-filtered. No Exchange change made for Shannon.
- Apex fix = add binding, not redirect. Restored apex by adding host-header bindings to the existing site (apex serves same content as www) — fastest restore; cert already SAN-covered apex so no cert change.
- HTTP→HTTPS redirect with ACME exclusion. Used IIS URL Rewrite (installed) with a
/.well-known/acme-challenge/negate condition so Certify/Let's Encrypt HTTP-01 renewal is not broken. 301 permanent. - Payment fix applied server-side first. Set the .NET TLS keys + app-pool recycle (the standard fix). Verified success via the payments DB rather than waiting on Tom; the app-code
ServicePointManager.SecurityProtocol = Tls12fallback was staged but not needed (DB confirmed live card approvals post-fix). - Security findings classified, never exfiltrated. All card/password determinations used aggregate
CASE/COUNTonly — no PAN or password value was selected or displayed. - #32378 left OPEN (In Progress) at Mike's direction for likely remediation work; billed but not resolved/closed.
- Tokenization is the card remediation — preserves the auto-pay feature while removing all stored PAN/CVV.
Problems Encountered
Get-MessageTraceV2not exposed viaadminapi/InvokeCommandinitially — investigator-exo token 401'd; the exchange-op token works and the cmdlet runs via InvokeCommand with a ≤10-day window constraint. The ExchangeOnlineManagement PowerShell module is NOT installed on GURU-5070 (used the REST InvokeCommand path instead).- PowerShell quoting/parse errors in several RMM probes (escaped quotes inside
$(...),foreach | Format-Tableempty-pipe, DBNull comparison). Resolved by collecting into variables, moving counts out of inline subexpressions, and DBNull-safe formatters. - Local apex test 404'd because the binding is IP-scoped to 192.168.8.72 and
localhostis 127.0.0.1 — verified externally instead (301/200/acme-404 all correct). - Syncro long-comment heredoc broke bash (apostrophes inside
$(cat <<EOF)). Resolved by writing the body to a file with a direct heredoc redirect, collapsing newlines to<br>, and posting viajq -n --rawfile.
Configuration Changes
On server WWW (192.168.8.72) — via GuruRMM:
- IIS site
glaztech_new: added bindingshttp/192.168.8.72:80:glaztech.comandhttps/192.168.8.72:443:glaztech.com(apex; reuses cert thumbprint2684b127f039be5cb5adea6f3f616c111824e4dc). D:\web\glaztech_4\web.config: inserted URL Rewrite rule "HTTP to HTTPS" (301, host-preserving, with/.well-known/acme-challenge/exclusion) as the first rule. Backup atD:\web\glaztech_4\web.config.bak-20260603-090701.- Registry:
SchUseStrongCrypto=1+SystemDefaultTlsVersions=1(DWORD) added to BOTHHKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319andHKLM\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v4.0.30319; recycled app poolglaztech_new.
Repo (ClaudeTools):
- Created
clients/glaztech/reports/2026-06-03-pci-cardholder-data-finding.md - Created
clients/glaztech/reports/2026-06-03-website-security-assessment.md(incl. attack-path + card data-flow sections)
Syncro tickets: #32376 (apex 404 + redirect), #32377 (payment TLS), #32378 (security assessment — left open).
Credentials & Secrets
- MailProtector / CloudFilter partner portal — NO credentials in the vault. ACG manages it via manual partner-portal login. (Action: consider vaulting for future automation.)
- MSP360 API (unrelated, used 6/2):
msp-tools/msp360-api.sops.yaml. - Exchange Operator app (message trace): vault
msp-tools/computerguru-exchange-operator.sops.yaml, appb43e7342-5b4b-492f-890f-bb5a4f7f40e9, tenant82931e3c-de7a-4f74-87f7-fe714be1f160; token viaget-token.sh <tenant> exchange-op. - Glaztech SQL (observed in site Web.config, NOT vaulted): server
192.168.8.62,3436, logintom(password inD:\web\glaztech_4\Web.configconnectionStrings), 15 per-office databases (glaz_prod,glaz_prod_phx, …,glaz_prod_corp) +mas_gti(Sage 100) +qqest(TimeForce).
Infrastructure & Servers
- WWW — Windows Server 2019 Standard, build 10.0.17763.8755 (patched through May 2026). Local IP 192.168.8.72, public 65.113.52.88. IIS 10.0, .NET 4.8. Site
glaztech_new→D:\web\glaztech_4(full VB.NET source on disk, not precompiled). LE certCN=www.glaztech.com, SAN glaztech.com + www, expires 2026-08-19, via Certify The Web (HTTP-01). GuruRMM agent id455a1bc7-1c29-42bc-b597-fa1e64f08eec. - SQL —
192.168.8.62,3436, 15 office DBs + Sage 100 (mas_gti) + TimeForce (qqest). - Payment gateway — CyberSource REST (
api.cybersource.com/pts/v2/payments) as PNC merchant processor (card path); CyberSource SOAP toolkit for eCheck/ACH (cybs.log). - Email/spam — MailProtector inbound filter
glaztech-com.inbound.emailservice.io; digests fromnoreply@azcomputerguru.com.
Commands & Outputs
- Apex 404:
curl -I https://glaztech.com→ 404Microsoft-HTTPAPI/2.0; post-fix → 200;/.well-known/acme-challenge/probeover http → 404 plain (renewal safe). - Payment TLS test (WWW → api.cybersource.com): TLS1.0/1.1 FAIL (SSPI), TLS1.2 OK; post-fix DB shows credit-card approval id 84455 at 09:36 (
web_payment_header, glaz_prod_corp). - Auth proc
get_web_accesslevel:where acct_no=@username and web_password=@passwd(plaintext password compare). quo():Return "'" + stext + "'"(no escaping → SQLi).get_cc_data:select * from cc_file where acct_no=@acctno(returns full PAN+CVV; IDOR-shaped).- Card storage classification:
cc_file~780 rows total (plaintext PAN, 0 encrypted);cc_file.cc_code50/54 CVV in Tucson;cof_payments_headerPhoenix 14,496 rows / 11,794 plaintext PAN;web_security~9,000+ plaintext passwords (corp 6,017 + tuc 3,012), 0 hash-like. - Message trace (Shannon): 629 noreply digests in 10 days to ~60 recipients, 0 to shannon@glaztech.com.
Pending / Incomplete Tasks
- MailProtector: provision/enable Shannon's Spam Summary in the MailProtector partner portal (manual). The earlier
gtimail@glaztech.comdaily digest failure (Priority-1 reject rule) is still open with Steve. - #32378 remediation (open ticket): quick wins ready on Mike's go — purge stored CVV (
cc_file.cc_code, backup-first, needs explicit sign-off), setdebug="false"+customErrors="On", remove RealVNC 4.2.8 + stale ScreenConnect, disable TLS 1.0/1.1. Then: hash passwords + stop emailing them, parameterize payment-page SQL (killquo()), tokenize card-on-file (CyberSource vault), separate dev from prod, least-privilege DB. - Payment fix: monitor that card successes continue; Tom code fallback (
SecurityProtocol = Tls12) staged if needed. - Backups contain plaintext cards — cleanup must address DB backups, not just live data.
Reference Information
- Tickets: #32376 (id 112107224, apex 404+redirect, Resolved, 1h remote), #32377 (id 112109440, CyberSource TLS, Resolved, 1.5h emergency remote), #32378 (id 112111185, security assessment, In Progress, 1h remote).
- Glaztech Syncro customer id 143932; prepaid block 26.5 → 22.25 hrs today.
- Reports:
clients/glaztech/reports/2026-06-03-pci-cardholder-data-finding.md,…-website-security-assessment.md. - GuruRMM client id
d857708c-5713-4ee5-a314-679f86d2f9f9; WWW agent455a1bc7-1c29-42bc-b597-fa1e64f08eec. - TLS fix pattern (legacy .NET → modern gateway):
SchUseStrongCrypto=1+SystemDefaultTlsVersions=1both.NETFramework\v4.0.30319hives + app-pool recycle. - LE-safe HTTP→HTTPS redirect: URL Rewrite rule with
{REQUEST_URI}^/\.well-known/acme-challenge/negate condition.