azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
10e8d7b6bbe3f1a4b5b167cce445692974c8f5f7
claudetools/clients/glaztech/session-logs/2026-06-03-session.md
Mike Swanson 10e8d7b6bb sync: auto-sync from GURU-5070 at 2026-06-03 19:39:32 
Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-06-03 19:39:32
2026-06-03 19:39:36 -07:00

17 KiB
Raw Blame History

Session Log — 2026-06-03 — Glaz-Tech Industries

User

  • User: Mike Swanson (mike)
  • Machine: GURU-5070
  • Role: admin

Session Summary

Worked a series of Glaztech web/payment issues on server WWW (192.168.8.72 / public 65.113.52.88), all via the GuruRMM agent, ending in a full security assessment of the website and its databases.

First, diagnosed why shannon@glaztech.com was not receiving MailProtector quarantine digests (sent as noreply@azcomputerguru.com). A Get-MessageTraceV2 (Exchange Operator app, tenant 82931e3c) over the last 10 days showed 629 digests delivered to ~60 recipients and zero to Shannon. Confirmed Shannon is a real, active mailbox receiving normal mail. Conclusion: not an Exchange/EOP problem (the morning SCL-bypass rule is irrelevant to her) — MailProtector is not generating a digest for her; the fix is on the MailProtector side (provision her / enable the Spam Summary). Vault has no MailProtector credentials, so it is a manual partner-portal task.

Handled an emergency apex 404: glaztech.com returned 404 (HTTP.sys) while www.glaztech.com returned 200, because IIS site glaztech_new had a host-header binding only for www. Added apex bindings (http:80 + https:443, reusing the existing SAN cert that already covers both names). Then added a URL Rewrite HTTP→HTTPS 301 redirect with a /.well-known/acme-challenge/ exclusion so Let's Encrypt (Certify The Web, HTTP-01) renewal is unaffected. Created and invoiced ticket #32376 (1h remote, prepaid).

Diagnosed and fixed the CyberSource credit-card payment outage ("Could not create SSL/TLS secure channel," 20 attempts / 0 success across all offices). Root cause: CyberSource (PNC processor) disabled TLS 1.0/1.1; the .NET payment library on Server 2019 negotiated old TLS. Applied SchUseStrongCrypto=1 + SystemDefaultTlsVersions=1 to both .NETFramework\v4.0.30319 hives and recycled the glaztech_new app pool. Verified via the payments DB that card payments resumed (a real credit-card approval at 09:36 after the fix; eCheck path was never affected). Created and invoiced ticket #32377 (1.5h emergency remote, prepaid).

Performed a deep security assessment (read-only; no card numbers or passwords retrieved). Findings are severe: full PANs and CVV stored in plaintext, ~9,000+ plaintext passwords, SQL injection via a fake quo() escaper in payment pages, reflected XSS, debug/error disclosure, the production server doubling as a dev workstation, remote-access sprawl (incl. RealVNC 4.2.8), and a listener accepting TLS 1.0/1.1. Produced two reports and a high-priority client ticket (#32378) with a plain-language, point-by-point public writeup for Steve/Tom. Also traced why cards are stored (card-on-file invoice auto-pay via gt_auto_process_2020.dll) and confirmed the exposure is contained to the web app's databases (not Sage 100). Billed 1h remote to #32378 and left it open for remediation.

Key Decisions

  • MailProtector digest = MailProtector-side issue. Message trace proved Shannon gets no digest at all; the EOP SCL-bypass rule (created earlier) only helps users whose digests were being spam-filtered. No Exchange change made for Shannon.
  • Apex fix = add binding, not redirect. Restored apex by adding host-header bindings to the existing site (apex serves same content as www) — fastest restore; cert already SAN-covered apex so no cert change.
  • HTTP→HTTPS redirect with ACME exclusion. Used IIS URL Rewrite (installed) with a /.well-known/acme-challenge/ negate condition so Certify/Let's Encrypt HTTP-01 renewal is not broken. 301 permanent.
  • Payment fix applied server-side first. Set the .NET TLS keys + app-pool recycle (the standard fix). Verified success via the payments DB rather than waiting on Tom; the app-code ServicePointManager.SecurityProtocol = Tls12 fallback was staged but not needed (DB confirmed live card approvals post-fix).
  • Security findings classified, never exfiltrated. All card/password determinations used aggregate CASE/COUNT only — no PAN or password value was selected or displayed.
  • #32378 left OPEN (In Progress) at Mike's direction for likely remediation work; billed but not resolved/closed.
  • Tokenization is the card remediation — preserves the auto-pay feature while removing all stored PAN/CVV.

Problems Encountered

  • Get-MessageTraceV2 not exposed via adminapi/InvokeCommand initially — investigator-exo token 401'd; the exchange-op token works and the cmdlet runs via InvokeCommand with a ≤10-day window constraint. The ExchangeOnlineManagement PowerShell module is NOT installed on GURU-5070 (used the REST InvokeCommand path instead).
  • PowerShell quoting/parse errors in several RMM probes (escaped quotes inside $(...), foreach | Format-Table empty-pipe, DBNull comparison). Resolved by collecting into variables, moving counts out of inline subexpressions, and DBNull-safe formatters.
  • Local apex test 404'd because the binding is IP-scoped to 192.168.8.72 and localhost is 127.0.0.1 — verified externally instead (301/200/acme-404 all correct).
  • Syncro long-comment heredoc broke bash (apostrophes inside $(cat <<EOF)). Resolved by writing the body to a file with a direct heredoc redirect, collapsing newlines to <br>, and posting via jq -n --rawfile.

Configuration Changes

On server WWW (192.168.8.72) — via GuruRMM:

  • IIS site glaztech_new: added bindings http/192.168.8.72:80:glaztech.com and https/192.168.8.72:443:glaztech.com (apex; reuses cert thumbprint 2684b127f039be5cb5adea6f3f616c111824e4dc).
  • D:\web\glaztech_4\web.config: inserted URL Rewrite rule "HTTP to HTTPS" (301, host-preserving, with /.well-known/acme-challenge/ exclusion) as the first rule. Backup at D:\web\glaztech_4\web.config.bak-20260603-090701.
  • Registry: SchUseStrongCrypto=1 + SystemDefaultTlsVersions=1 (DWORD) added to BOTH HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319 and HKLM\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v4.0.30319; recycled app pool glaztech_new.

Repo (ClaudeTools):

  • Created clients/glaztech/reports/2026-06-03-pci-cardholder-data-finding.md
  • Created clients/glaztech/reports/2026-06-03-website-security-assessment.md (incl. attack-path + card data-flow sections)

Syncro tickets: #32376 (apex 404 + redirect), #32377 (payment TLS), #32378 (security assessment — left open).

Credentials & Secrets

  • MailProtector / CloudFilter partner portal — NO credentials in the vault. ACG manages it via manual partner-portal login. (Action: consider vaulting for future automation.)
  • MSP360 API (unrelated, used 6/2): msp-tools/msp360-api.sops.yaml.
  • Exchange Operator app (message trace): vault msp-tools/computerguru-exchange-operator.sops.yaml, app b43e7342-5b4b-492f-890f-bb5a4f7f40e9, tenant 82931e3c-de7a-4f74-87f7-fe714be1f160; token via get-token.sh <tenant> exchange-op.
  • Glaztech SQL (observed in site Web.config, NOT vaulted): server 192.168.8.62,3436, login tom (password in D:\web\glaztech_4\Web.config connectionStrings), 15 per-office databases (glaz_prod, glaz_prod_phx, …, glaz_prod_corp) + mas_gti (Sage 100) + qqest (TimeForce).

Infrastructure & Servers

  • WWW — Windows Server 2019 Standard, build 10.0.17763.8755 (patched through May 2026). Local IP 192.168.8.72, public 65.113.52.88. IIS 10.0, .NET 4.8. Site glaztech_new → D:\web\glaztech_4 (full VB.NET source on disk, not precompiled). LE cert CN=www.glaztech.com, SAN glaztech.com + www, expires 2026-08-19, via Certify The Web (HTTP-01). GuruRMM agent id 455a1bc7-1c29-42bc-b597-fa1e64f08eec.
  • SQL — 192.168.8.62,3436, 15 office DBs + Sage 100 (mas_gti) + TimeForce (qqest).
  • Payment gateway — CyberSource REST (api.cybersource.com/pts/v2/payments) as PNC merchant processor (card path); CyberSource SOAP toolkit for eCheck/ACH (cybs.log).
  • Email/spam — MailProtector inbound filter glaztech-com.inbound.emailservice.io; digests from noreply@azcomputerguru.com.

Commands & Outputs

  • Apex 404: curl -I https://glaztech.com → 404 Microsoft-HTTPAPI/2.0; post-fix → 200; /.well-known/acme-challenge/probe over http → 404 plain (renewal safe).
  • Payment TLS test (WWW → api.cybersource.com): TLS1.0/1.1 FAIL (SSPI), TLS1.2 OK; post-fix DB shows credit-card approval id 84455 at 09:36 (web_payment_header, glaz_prod_corp).
  • Auth proc get_web_accesslevel: where acct_no=@username and web_password=@passwd (plaintext password compare).
  • quo(): Return "'" + stext + "'" (no escaping → SQLi).
  • get_cc_data: select * from cc_file where acct_no=@acctno (returns full PAN+CVV; IDOR-shaped).
  • Card storage classification: cc_file ~780 rows total (plaintext PAN, 0 encrypted); cc_file.cc_code 50/54 CVV in Tucson; cof_payments_header Phoenix 14,496 rows / 11,794 plaintext PAN; web_security ~9,000+ plaintext passwords (corp 6,017 + tuc 3,012), 0 hash-like.
  • Message trace (Shannon): 629 noreply digests in 10 days to ~60 recipients, 0 to shannon@glaztech.com.

Pending / Incomplete Tasks

  • MailProtector: provision/enable Shannon's Spam Summary in the MailProtector partner portal (manual). The earlier gtimail@glaztech.com daily digest failure (Priority-1 reject rule) is still open with Steve.
  • #32378 remediation (open ticket): quick wins ready on Mike's go — purge stored CVV (cc_file.cc_code, backup-first, needs explicit sign-off), set debug="false" + customErrors="On", remove RealVNC 4.2.8 + stale ScreenConnect, disable TLS 1.0/1.1. Then: hash passwords + stop emailing them, parameterize payment-page SQL (kill quo()), tokenize card-on-file (CyberSource vault), separate dev from prod, least-privilege DB.
  • Payment fix: monitor that card successes continue; Tom code fallback (SecurityProtocol = Tls12) staged if needed.
  • Backups contain plaintext cards — cleanup must address DB backups, not just live data.

Reference Information

  • Tickets: #32376 (id 112107224, apex 404+redirect, Resolved, 1h remote), #32377 (id 112109440, CyberSource TLS, Resolved, 1.5h emergency remote), #32378 (id 112111185, security assessment, In Progress, 1h remote).
  • Glaztech Syncro customer id 143932; prepaid block 26.5 → 22.25 hrs today.
  • Reports: clients/glaztech/reports/2026-06-03-pci-cardholder-data-finding.md, …-website-security-assessment.md.
  • GuruRMM client id d857708c-5713-4ee5-a314-679f86d2f9f9; WWW agent 455a1bc7-1c29-42bc-b597-fa1e64f08eec.
  • TLS fix pattern (legacy .NET → modern gateway): SchUseStrongCrypto=1 + SystemDefaultTlsVersions=1 both .NETFramework\v4.0.30319 hives + app-pool recycle.
  • LE-safe HTTP→HTTPS redirect: URL Rewrite rule with {REQUEST_URI} ^/\.well-known/acme-challenge/ negate condition.

Update: 15:52 PT — Payrilla/CyberSource reconciliation + Sage finding

Mike reported Glaztech "no longer uses CyberSource, switched to Payrilla (secure card storage)." Verified the live system rather than take it at face value — and the technical reality contradicts the belief:

  • Website is STILL on CyberSource/PNC. No payrilla/payrix/paya/payroc references anywhere in D:\web\glaztech_4 code or config. Live payment pages remain online-payment-pnc.aspx / quick-pay-pnc.aspx → api.cybersource.com. web_payment_header shows live CC-WebPayment-PNC Approved/Error transactions today (2026-06-03 14:47 Approved, 15:07 Error) + E-Check.
  • Plaintext cards STILL being written daily. cc_file last write: Tucson 2026-06-03 14:15 (8/60d), Phoenix 2026-06-03 10:19 (14/60d). Ongoing accumulation, not legacy.
  • Sage 100 (mas_gti) CC module is DISABLED — SY_Company (Company GTI) CreditCardEnable=N, CreditCardVaultOnlyID empty, AR_CustomerCreditCard = 0 rows. Corrects an earlier in-session assumption that Sage actively tokenizes via Paya: the tokenization columns exist in the Sage schema but are unused. Sage stores no cardholder data at all (strengthens the containment finding).
  • Payrilla/Paya is not visible in either system we can reach. If in use, it's a separate channel (standalone virtual terminal / different portal / manual) that does NOT cover the website. This is the likely source of the client's "we moved to Payrilla for everything" belief — true at most for some standalone tool, NOT for the website (the system storing plaintext cards).
  • Anomaly: cc_file now returns "Invalid object name" in the corp DB though it existed there this morning (3 rows) — needs a second look; per-office DBs (tuc/phx) unaffected and still writing.

Actions: updated clients/glaztech/reports/2026-06-03-website-security-assessment.md (added "Current State Verified — 2026-06-03" section; corrected two Sage lines from "uses tokenization / materially compliant" to "CC module disabled, 0 stored cards"). Posted public+emailed client comment on #32378 (id 417040624) reflecting the live, ongoing risk and that Payrilla is not yet implemented on the site. Drafted a further client clarification (the Sage/Payrilla visibility gap) — pending Mike's go before posting (would be the 4th #32378 email today).

Open: confirm with Payrilla which flows actually route through them; investigate the corp cc_file anomaly; remediation path reframed to "migrate the website's card-on-file to the chosen processor's token vault, stop writing cc_file/cof_payments_header, purge plaintext + backups, decommission CyberSource."

Update: 19:32 PT — Tom (dev) reply thread: storage = GTIware, website = sysadmin access vector (TOP CRITICAL)

Tom (Glaztech's internal IT / author of the GTIware/GlazGTI PSA) replied to #32378 asking where the website stores data, stating he stores nothing from the website's online payment system. Investigated and Tom is correct — and it surfaced the single worst finding of the engagement.

  • The website stores NO cards. Zero cc_file/save_cc_data references in the site's .aspx/.vb; the quick-pay pages even carry a disclaimer "Glaz-Tech Industries does not save nor record any card info." The site uses the shared gt_utilities namespace (209 refs) for general plumbing only.
  • The cards are written by GTIware (Tom's PSA): the cc_file/save_cc_data logic is in compiled libraries glaztech_utilities_2020.dll + gt_auto_process_2020.dll (deployed in the site's Bin but not called by the site). Recent cc_file rows are stamped with staff usernames (Victoria, Bryce, Diana) and notes like "RUN CARD WHEN REQUESTED" — a back-office save-a-card workflow.
  • TOP CRITICAL (now C0 in the report): the website connects to the GTIware SQL server as a sysadmin. The site's Web.config uses SQL auth user id=tom (login id 267, a named SQL login, member of the sysadmin role — NOT the built-in sa, created 2018; password embedded in Web.config). The SQL host is GTI-INV-SQL (192.168.8.62,3436), shared by the website and GTIware, with 46 databases (all offices' glaz_prod* + *_archive + PDF stores + qqest TimeForce + gti_samsara + system DBs). Because SQLi runs as the connecting login, the website's quo() SQL-injection executes as sysadmin. Cross-DB confirmed live: from the Tucson connection, read other offices' glaz_prod_phx.dbo.cc_file (141), glaz_prod_den (190), glaz_prod_elp (179). So one guessed website login + the SQLi = full read/write/destroy of the entire GTIware instance + xp_cmdshell OS takeover of the SQL server.
  • get_cc_data is SELECT * FROM cc_file WHERE acct_no=@acctno → returns full PAN+CVV; IDOR-shaped on @acctno. Other sysadmin logins on the instance: GTI-INV-SQL\Administrator, NT SERVICE\*, sa (enabled), tom.

Actions:

  • Added C0 (top critical) to the website security assessment with the full attack tree (steal/destroy/manipulate/OS-takeover) and required remediation (least-privilege web login that cannot see GTIware data; separate/partition the website DB from GTIware; stop storing cards — call processor API directly or tokenize; never store CVV (PCI 3.2); encrypt at rest; fix SQLi + lockout).
  • Corrected both reports' attribution: storage = GTIware internal PSA (staff-operated), website = access path; Sage CC module disabled (0 cards), not a CHD location.
  • Posted plain-English reply to Tom on #32378 (public+emailed, comment 417070212): leads with "how we found it" (a website-only review reached internal GTIware card data → that reachability IS the flaw), explains the sysadmin+SQLi chain plainly, lists the four fixes.
  • #32378 remains Waiting on Customer.

Open / next: Tom to remediate (pull sysadmin login from the site → least-privilege; isolate website DB from GTIware; tokenize / stop storing; never store CVV; fix SQLi + lockout). The corp cc_file "Invalid object name" anomaly still unexplained. ACG can execute the quick wins (CVV purge, least-privilege login, debug-off) on request.

Reference in New Issue View Git Blame Copy Permalink