sync: auto-sync from GURU-5070 at 2026-06-03 11:52:45

Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-06-03 11:52:45
This commit is contained in:
2026-06-03 11:52:50 -07:00
parent 6228793152
commit 6de0ce6098
45 changed files with 1452 additions and 35 deletions

View File

@@ -0,0 +1,89 @@
# Glaz-Tech Industries — Cardholder Data Storage Finding (PCI-DSS)
**Classification:** CONFIDENTIAL — Security / PCI
**Date:** 2026-06-03
**Prepared by:** Arizona Computer Guru (Mike Swanson)
**System:** Glaztech e-commerce/customer-portal web application (server `WWW`, 192.168.8.72) and its SQL Server backend (`192.168.8.62,3436`)
**Method:** Read-only database inspection via GuruRMM. **No actual card numbers were retrieved.** All determinations were made by column **classification and aggregate counts** (length, all-digits vs. masked vs. encrypted patterns). No PAN, CVV, or cardholder value is reproduced in this report.
---
## Executive Summary
The custom Glaztech web application stores **full credit card numbers (PAN) in plaintext** and **retains card security codes (CVV/CID)** in its SQL databases. This is a serious PCI-DSS compliance failure and a significant breach-liability exposure. Approximately **780 saved customer cards** (card-on-file) and **tens of thousands of historical payment records** are affected, with confirmed plaintext PANs and ~hundreds of stored CVV values.
The Sage 100/MAS ERP database (`mas_gti`) is **not** the problem — it uses proper tokenization (`CreditCardGUID`, `Last4Unencrypted`, `EncryptedCreditCardNo`). **The exposure is entirely in the custom web application's own tables.**
---
## Findings
### Finding 1 — Full PAN stored in plaintext (card-on-file) — HIGH/CRITICAL
- **Table:** `dbo.cc_file` (`cc_number varchar(50)`, `cc_exp`, `cc_name`, `cc_code`, `activate`, …), present in every per-office database.
- **Encryption:** none. Values classify as all-digits or digits-with-spaces/dashes (recoverable PAN). Zero rows contained alpha/base64 (encrypted) content.
- **Scale (`cc_file` row counts by office DB):**
| DB | rows | DB | rows |
|---|---|---|---|
| glaz_prod_elp | 179 | glaz_prod_alb | 44 |
| glaz_prod_den | 190 | glaz_prod_slc | 41 |
| glaz_prod_phx | 141 | glaz_prod_boi | 26 |
| glaz_prod_brl | 88 | glaz_prod_shp | 14 |
| glaz_prod (tuc) | 54 | glaz_prod_corp | 3 |
| **Total** | | | **≈ 780 saved cards** |
- Sample classification (`glaz_prod` / tuc, 54 rows): 29 plaintext PAN + 25 formatted PAN, **0 encrypted**, length 1520.
- **PCI-DSS Req 3.4:** PAN must be rendered unreadable anywhere it is stored. **Violated.**
### Finding 2 — Stored CVV / card security codes — CRITICAL (indefensible)
- **Column:** `dbo.cc_file.cc_code` (`varchar(20)`).
- **Result:** populated with 34 digit values. In `glaz_prod` (tuc): **50 of 54 rows hold a stored CVV** (34 numeric digits). Other office DBs expected to mirror this.
- **PCI-DSS Req 3.2:** sensitive authentication data (CVV/CVV2/CID) must **never** be stored after authorization — **no exception, even encrypted.** This is the most serious item and the fastest to remediate.
### Finding 3 — Plaintext PAN in historical payment records — HIGH (large volume)
- **Table:** `dbo.cof_payments_header` (`cc_number varchar(50)`), per-office.
- **Scale / classification:**
- `glaz_prod_phx`: **14,496 rows — 11,794 full plaintext PANs**, ~2,700 masked/other.
- `glaz_prod` (tuc): 2,245 rows — 367 plaintext + 597 formatted + ~1,242 masked/other.
- Represents years of transaction history retained with recoverable card numbers.
### Finding 4 — Access controls / context
- The web application connects with a **single shared SQL login (`tom`)** that has **full read on the card columns** (verified: `HAS_PERMS_BY_NAME` = 1 on `cc_number` and `cc_code`). No column-level protection or data masking.
- Connection strings (with this login + password) are stored in the site `Web.config` on the web server.
- The Sage 100 DB (`mas_gti`) uses tokenization and is materially compliant by comparison — scope of this finding is the **custom web app** only.
---
## Risk & Impact
- **Breach exposure:** if the SQL server or web server is compromised, this is a reportable cardholder-data breach (full PAN + CVV in the clear).
- **Compliance:** clear violations of PCI-DSS Req 3.2 (CVV retention) and Req 3.4 (PAN protection); likely invalidates the merchant's self-assessment and exposes Glaztech to card-brand fines and liability.
- **CVV retention** specifically removes most "we were compliant" defenses in the event of a breach.
---
## Remediation Plan
**Immediate (days):**
1. **Stop writing `cc_code` (CVV)** in the application and **purge all stored CVV values** (`cc_file.cc_code`). This is the highest-priority, lowest-effort risk reduction and is indefensible to retain.
2. Inventory and restrict who/what can read `cc_file` / `cof_payments_header`.
**Short term (weeks):**
3. Stop storing raw PAN. Move card-on-file to the **CyberSource token vault** (already in use for processing) — store a token reference instead of the PAN.
4. Purge or strongly encrypt the historical `cc_number` columns in `cof_payments_header` and `cc_file` once tokenization is in place.
**Hardening:**
5. Least-privilege SQL accounts (separate read/write, no blanket card-column read), encryption at rest (TDE) as defense-in-depth, and secrets management for `Web.config` connection strings.
6. Re-scope/refresh the merchant PCI SAQ after remediation.
---
## Evidence / Methodology (PAN-free)
- Column discovery via `INFORMATION_SCHEMA.COLUMNS` (names matching card/cvv/expir/etc.).
- Storage classification via aggregate `CASE`/`COUNT` only — e.g. `SUM(CASE WHEN col NOT LIKE '%[^0-9]%' AND LEN(col) BETWEEN 13 AND 19 THEN 1 ELSE 0 END)` to count full-PAN-shaped values without selecting them.
- CVV detection via count of 34 digit `cc_code` values. No raw card data was selected, displayed, or stored at any point.
---
**Status:** Reported to ownership 2026-06-03. **No data was modified or deleted** — remediation pending client (Steve Eastman / Tom) direction. This finding should be raised with the client as a priority compliance/security item.

View File

@@ -0,0 +1,154 @@
# Glaz-Tech Industries — Website Security Assessment
**Classification:** CONFIDENTIAL — Security
**Date:** 2026-06-03
**Assessor:** Arizona Computer Guru (Mike Swanson)
**Target:** Glaztech customer/e-commerce web application — server `WWW` (192.168.8.72, public 65.113.52.88), site `glaztech_new` at `D:\web\glaztech_4`, SQL backend `192.168.8.62,3436`
**Method:** Authorized read-only assessment via GuruRMM (config/registry inspection, source-code review of the on-server VB.NET source, and read-only/aggregate DB inspection). **No cardholder data and no passwords were retrieved** — sensitive columns were classified by aggregate only.
> Companion report: `2026-06-03-pci-cardholder-data-finding.md` (cardholder-data storage detail).
---
## Overall Risk: CRITICAL
The site stores cardholder data (PAN + CVV) and **all user passwords in plaintext**, contains SQL injection and reflected XSS, and runs on a server that doubles as a developer workstation with extensive remote-access and end-of-life software. Multiple findings are independently sufficient to cause a reportable breach.
| # | Finding | Severity |
|---|---|---|
| C1 | Plaintext PAN + stored CVV in DB | Critical |
| C2 | All user passwords stored in plaintext; passwords emailed in cleartext | Critical |
| C3 | SQL injection via fake `quo()` escaper (incl. payment pages) | Critical |
| C4 | Reflected XSS in `gt_errorpage.aspx` | High |
| H1 | Production payment server is also a dev workstation (VS, SDKs, build tools) | High |
| H2 | Remote-access sprawl incl. end-of-life RealVNC 4.2.8 + stale ScreenConnect v6 | High |
| H3 | `debug="true"` + `customErrors=Off` + exceptions echoed to users | High |
| H4 | Server accepts TLS 1.0/1.1 on the listener | High |
| H5 | No cookie Secure/HttpOnly hardening, no MFA, no lockout, session-fixation risk | High |
| H6 | Single shared SQL login with full card-column read; creds in `Web.config` | High |
| M1 | Outdated/unused third-party components; SHA1 machineKey; source on prod | Medium |
---
## Critical Findings
### C1 — Plaintext PAN and stored CVV (see companion PCI report)
`cc_file` (~780 saved cards) and `cof_payments_header` (tens of thousands of rows; e.g. Phoenix 14,496 / 11,794 plaintext) store full card numbers unencrypted, and `cc_file.cc_code` retains CVV (PCI Req 3.2 — prohibited). Detail and remediation in the PCI report.
### C2 — All passwords stored in plaintext; cleartext password email
- **Customer portal:** auth stored proc `get_web_accesslevel` compares `web_security.web_password = @passwd` **with no hashing**. `web_security` holds **~9,000+ plaintext passwords** (corp 6,017 + tuc 3,012 + other offices), 0 hash-like values, lengths 319.
- **Employees:** `emp/employee-login.aspx` "forgot password" verifies last name + email, then **emails the user their existing plaintext password** (`"The password to your employee profile is: " + pword`) — only possible with reversible/plaintext storage.
- **Impact:** any DB read (or the existing SQLi) exposes every customer/employee credential in the clear; password reuse means broad downstream compromise. Weak "lastname + email" knowledge check gates the password email.
- **Fix:** store only salted password hashes (PBKDF2/bcrypt/Argon2); never email passwords — implement a reset-token flow; force a global password reset after remediation.
### C3 — SQL injection via non-escaping `quo()` helper
```vb
Function quo(stext) As String
Return "'" + stext + "'" ' wraps in quotes but does NOT escape embedded quotes
End Function
```
- Used to build **concatenated dynamic SQL** in multiple pages including payment flows (`ach.aspx.vb`, `quick-pay-ach.aspx.vb`, `quick-pay-pnc.aspx.vb`, `quick-pay.aspx.vb`, `order-detail*`). Any input containing `'` breaks out of the string → injection.
- Codebase posture is *mixed*: 948 properly parameterized calls vs. **59 concatenated SQL statements** (~10 joining user input). The login path itself is parameterized (sproc) and **not** injectable; the risk is the concatenated set.
- **Fix:** replace all concatenation with parameterized commands / stored procedures; delete `quo()`. Prioritize payment pages.
### C4 — Reflected XSS in `gt_errorpage.aspx`
- `smessage = Request.QueryString("errmsg")` (line 20) → `lblerr.Text = smessage` (line 48). `Label.Text` is **not** HTML-encoded, and the app redirects many exceptions to `gt_errorpage.aspx?errmsg=<msg>` (often containing raw `ex.Message`). An attacker-supplied `errmsg=<script>…</script>` executes in the victim's browser.
- **Fix:** HTML-encode (`Server.HtmlEncode`) before output; stop placing exception text in URLs; show generic errors to users and log details server-side.
---
## Attack Path — A Single Guessed Login → the Entire Card Database
Chaining the findings into the realistic worst case, with difficulty ratings.
**Step 1 — Obtain a customer login (LOW).** Username = the customer **account number** (enumerable, not secret). Passwords are plaintext, **as short as 3 characters**, no complexity rules, and there is **no account lockout or rate-limiting** — unlimited guessing / credential-stuffing.
**Step 2 — Normal UI (masked display).** Payment pages *display* cards masked to last-4 (`xxxx-xxxx-xxxx-1234`), so a point-and-click attacker sees last-4 + expiry + cardholder/billing data and can transact on saved cards. Note: the read proc `get_cc_data` is `SELECT * FROM cc_file WHERE acct_no=@acctno` — it returns the **full PAN and CVV to the application server**; only the *display* is masked, and the `@acctno` parameter makes it an IDOR-shaped full-card read. Any endpoint returning that proc's output unmasked (or the SQLi below) yields full numbers.
**Step 3 — SQL injection (FULL exposure).** The post-login payment pages (`quick-pay`, `ach`, `quick-pay-pnc`) build SQL with the non-escaping `quo()` helper and require only a valid session. A logged-in attacker can UNION-inject `SELECT cc_number, cc_code FROM cc_file` and exfiltrate **every stored full card number AND CVV** for the office — directly, because the data is **plaintext** (no encryption/key to defeat). UI masking is irrelevant at this layer.
| Goal | Difficulty |
|---|---|
| Obtain a valid login | **Low** (no lockout, guessable username, 3-char plaintext passwords) |
| See last-4 / transact via UI | **Low** |
| Exfiltrate ALL full PAN + CVV | **LowModerate** (one login + standard SQLi; plaintext data) |
**There is no defense-in-depth** — every compensating control (lockout, password hashing, PAN encryption, parameterized queries) is absent, so the first failure is the last failure. Highest-leverage breakers: login lockout/rate-limiting, parameterize the payment-page SQL (remove `quo()`), purge CVV + tokenize/encrypt PAN.
---
## Why the Cards Are Stored, and Where They Flow
**Business purpose — card-on-file invoice auto-pay.** Cards are stored (with an `activate` flag on `cc_file`) so the business can automatically charge customers' open invoices. The proc `i_get_cc_on_file_invoices` joins `invoice` × `cc_file` for active cards with an outstanding, delivered balance; `gt_auto_process_2020.dll` (in `bin`, with stale copies in `Old_bin`/`Old_code\Bin`) is the engine that reads the stored card and bills it via CyberSource. There is no scheduled task — the run is triggered from within the web app (most likely staff-initiated). The large `cof_payments_header` history (e.g. 14,496 rows in Phoenix) is years of these charges.
**Where the full PAN is used.** Only five DB objects reference the full `cc_number`: `save_cc_data`/`save_cc_data1`/`save_cc_data2` (writes) and the `is_cc_active`/`is_cc_on_file` functions. However, **`get_cc_data` is `SELECT *`**, so it also returns the full PAN + CVV whenever a saved card is read for charging — the full number crosses to the app server on every card-on-file charge; the UI only masks the *display*.
**Containment — does NOT spread to other systems.** The Sage 100 ERP DB (`mas_gti`) has **0** procedures referencing `cc_file` or `web_security` — the plaintext cards do **not** propagate into Sage (which uses tokenization). Exposure is **contained to the custom web app's 15 office databases** on SQL `192.168.8.62`. Secondary exposure surfaces: **database backups** (every backup of those DBs contains plaintext PAN + CVV) and **stale on-disk code/data copies** (`Old_bin`, `Old_code`).
**Fix preserves the feature.** Migrating card-on-file to the **CyberSource token vault** (store a token; let CyberSource hold the PAN) lets `gt_auto_process` keep auto-billing by token while removing every stored PAN/CVV — and removes the cardholder-data liability from backups.
---
## High Findings
### H1 — Production payment server is also a developer workstation
Installed on the live server: **Visual Studio Community 2015 and 2022, .NET 8 SDKs, MSBuild/Build Tools, TFS office integration, IIS 10 Express, Notepad++, WinRAR 7.22, OpenSSL 3.5.0**. Full application **source code is on the box** (128 `.vb` + 125 `.aspx.vb`, not precompiled). This massively expands attack surface and blast radius on the host that processes cardholder data. **Fix:** move development off the production host; deploy precompiled; remove SDKs/IDEs/dev tools.
### H2 — Remote-access sprawl, including end-of-life software
Present: **RealVNC Enterprise E4.2.8** (≈2009 — critically outdated, known auth-bypass-class issues), **ScreenConnect client v6.0.11622 (2018, stale)** alongside a current ScreenConnect, **Splashtop**, **Datto RMM + Datto EDR**, **Syncro**, plus GuruRMM. 6+ remote-management agents = large unmonitored access surface. **Fix:** remove RealVNC and the stale ScreenConnect immediately; rationalize to a single sanctioned remote-access tool; inventory who controls each.
### H3 — Debug/error information disclosure
`Web.config`: **`<compilation debug="true" …>`** in production and **`<customErrors mode="Off"/>`** present; login/employee code echo `ex.Message` to the page or via `errmsg`. Leaks stack traces, SQL errors, internal paths. **Fix:** `debug="false"`, `customErrors="On"` with a generic page, stop surfacing exception text to users.
### H4 — Listener accepts TLS 1.0/1.1
SChannel: **TLS 1.0 Server `Enabled=1`** (and TLS 1.1 at OS default = enabled); TLS 1.2 enabled. The public HTTPS endpoint therefore still negotiates deprecated TLS — a PCI listener finding. **Fix:** disable TLS 1.0/1.1 (SChannel server) after confirming no legacy client dependency; keep TLS 1.2.
### H5 — Session/credential handling
- Custom **Session-variable auth** (no ASP.NET Forms auth); **no session-ID regeneration on login** → session-fixation risk.
- **No `requireSSL`** and **no `httpOnlyCookies`** configured → cookies not marked Secure (site was HTTP-reachable until the 2026-06-03 HTTP→HTTPS redirect was added).
- **No MFA, no account lockout / rate limiting**; username = customer **account number** (guessable) → brute-force exposure.
- **Fix:** Secure+HttpOnly cookies, regenerate session on login, add lockout/throttling, consider MFA for employee/admin access.
### H6 — Database access model
Web app connects with a **single shared SQL login (`tom`)** that has full read on card and password columns (no column-level control); **connection strings with credentials are in `Web.config`** on the web server (15+ per-office DBs). **Fix:** least-privilege per-function accounts, remove blanket card/password read, protect/secret-manage connection strings, enable TDE at rest.
---
## Medium / Component Hygiene
- **Outdated third-party libraries in `bin`:** `AjaxControlToolkit 3.0.30930` (2008 — present but **not referenced**, remove it), `Microsoft.IdentityModel.Tokens` / `System.IdentityModel.Tokens.Jwt 5.1.2` (2017), `CyberSource SDK 1.4.10` (legacy), assorted GrapeCity ActiveReports versions. Inventory and update/remove.
- **`machineKey validation="SHA1"`** — move to SHA-256 (AES/HMACSHA256) with managed keys.
- **Source code resident on production** — remove; deploy build artifacts only.
- **OpenSSL 3.5.0 / WinRAR 7.22 / Chrome** on a server — patch or remove; reduce footprint.
---
## What is Acceptable (balanced view)
- **OS patching is current-ish:** Windows Server 2019, build 17763.8755, patched through **May 2026** (supported to 2029) — the OS itself is *not* the weak point.
- **Most data access is parameterized** (948 parameterized calls) — the SQLi exposure is a bounded set of concatenated queries, not pervasive.
- **The Sage 100 ERP DB** (`mas_gti`) uses proper tokenization (`CreditCardGUID`, `Last4Unencrypted`, `EncryptedCreditCardNo`) — materially compliant; the exposure is the **custom web app**, not Sage.
- TLS 1.2 to CyberSource now works (payment outage fixed 2026-06-03).
---
## Prioritized Remediation Roadmap
**Now (days):**
1. Purge stored CVV (`cc_file.cc_code`); stop writing it.
2. `debug="false"`, `customErrors="On"`; HTML-encode `gt_errorpage` output; stop echoing exceptions.
3. Remove RealVNC 4.2.8 and the stale ScreenConnect v6 client.
4. Disable TLS 1.0/1.1 on the listener.
**Short term (weeks):**
5. Convert passwords to salted hashes; replace the email-the-password flow with reset tokens; force a global reset.
6. Parameterize the concatenated SQL (payment pages first); delete `quo()`.
7. Secure+HttpOnly cookies; session regeneration; login throttling/lockout.
8. Move card-on-file to the CyberSource token vault; purge/encrypt historical PAN columns.
**Structural (project):**
9. Separate development from the production host; deploy precompiled; remove dev tooling and source from prod.
10. Least-privilege DB accounts, secret management for connection strings, TDE; re-scope the merchant PCI SAQ after remediation.
---
**Status:** Assessment complete 2026-06-03. **No changes were made to the application, database, or data during this assessment** (read-only). Findings to be reviewed with the client (Steve Eastman / Tom) as priority security and PCI remediation. This report contains no card numbers or passwords.

View File

@@ -0,0 +1,90 @@
# Session: 2026-06-03 - Remove SBS from mspbackups for Glaztech
**User:** Mike Swanson (mike)
**Machine:** GURU-5070
**Role:** admin
## Session Summary
Removed the SBS computer entry from MSP360/mspbackups for Glaztech Industries. This is the computer with MSP360 UserID d425fbbe-43f6-4fb7-8695-a9296b762a3b, ComputerName SBS, backup destination ACG-GLAZTECH (233GB used).
Followed locate-confirm-execute: used GrepAI, MSP360 API (as documented in 2026-06-01 session log), vault, B2 for verification (read-only).
Disabled the user via MSP360 API PUT /api/Users Enabled=false, then DELETE /api/Users/{id} (both 200).
This should trigger chained B2 data deletion for prefix MBS-d425fbbe-43f6-4fb7-8695-a9296b762a3b/CBB_SBS/ in ACG-GLAZTECH bucket. Did not touch B2 data directly.
## Key Decisions
- Removal only via MSP360 API (computer entry), per user clarification that license is expired, and removal triggers B2.
- Used MSP360 API with fresh token from /api/Provider/Login using vault creds.
- Claimed coord lock on clients/glaztech mspbackups/SBS.
- Snapshot: MSP360 user state (before/after Enabled), B2 files under prefix (still present), Monitoring still showed plan.
- No RMM agent found for SBS in glaztech (searches in code and API queries showed other GTI- machines).
- Documented in clients/glaztech/session-logs/
## Problems Encountered
- No active LicenseID found for SBS in current /api/Licenses (consistent with expired trial from report).
- MSP360 DELETE /api/Users/{id} succeeded 200 but entry still queryable with Enabled=false (soft delete/queue like vland example in docs).
- B2 deletion is asynchronous via MSP360 portal processing.
## Configuration Changes
- MSP360: User d425fbbe-43f6-4fb7-8695-a9296b762a3b set Enabled=false via PUT, then DELETE called.
## Commands & Outputs
(See tool calls in transcript for full API queries, vault, b2 lists, coord lock/release.)
Key:
- Vault get-field msp-tools/msp360-api.sops.yaml credentials -> login kY9PvDdWki password p9wzJFRT8nC6VfFz6UDZ
- Login POST -> token
- GET /api/Users -> found SBS entry
- GET /api/Monitoring -> showed plan for SBS, last backup 2026-05-08
- PUT /api/Users {"ID": "d425fbbe-43f6-4fb7-8695-a9296b762a3b", "Enabled": false} -> 200
- Re-GET user -> Enabled: false
- DELETE /api/Users/d425fbbe-43f6-4fb7-8695-a9296b762a3b -> 200
- B2: py ... b2.py files ACG-GLAZTECH --prefix MBS-d425fbbe-43f6-4fb7-8695-a9296b762a3b/CBB_SBS --limit 10 -> listed files (still there)
- Coord: POST lock, later DELETE release.
## Pending
- Monitor MSP360 portal / B2 for deletion progress of the SBS computer and its backup data in ACG-GLAZTECH.
- If RMM has mapping for glaztech SBS backup, reconcile (no current agent found).
- Update glaztech wiki/clients/glaztech.md if infrastructure changes documented.
## References
- MSP360 API usage: session-logs/2026-06-01-session.md (detailed commands, endpoints, vault, cross-ref)
- B2 mspbackups: .claude/skills/b2/SKILL.md
- Client context: wiki/clients/glaztech.md , clients/glaztech/
- Coord lock: released dcf3c54c-840a-4131-a653-69857bdeac45
- SPEC-023 for future automation: projects/msp-tools/guru-rmm/docs/specs/SPEC-023-msp360-license-release-on-decommission.md
## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin

View File

@@ -0,0 +1,106 @@
# Session Log — 2026-06-03 — Glaz-Tech Industries
## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin
---
## Session Summary
Worked a series of Glaztech web/payment issues on server `WWW` (192.168.8.72 / public 65.113.52.88), all via the GuruRMM agent, ending in a full security assessment of the website and its databases.
First, diagnosed why `shannon@glaztech.com` was not receiving MailProtector quarantine digests (sent as `noreply@azcomputerguru.com`). A `Get-MessageTraceV2` (Exchange Operator app, tenant 82931e3c) over the last 10 days showed 629 digests delivered to ~60 recipients and **zero to Shannon**. Confirmed Shannon is a real, active mailbox receiving normal mail. Conclusion: not an Exchange/EOP problem (the morning SCL-bypass rule is irrelevant to her) — MailProtector is not generating a digest for her; the fix is on the MailProtector side (provision her / enable the Spam Summary). Vault has no MailProtector credentials, so it is a manual partner-portal task.
Handled an emergency apex 404: `glaztech.com` returned 404 (HTTP.sys) while `www.glaztech.com` returned 200, because IIS site `glaztech_new` had a host-header binding only for `www`. Added apex bindings (http:80 + https:443, reusing the existing SAN cert that already covers both names). Then added a URL Rewrite HTTP→HTTPS 301 redirect with a `/.well-known/acme-challenge/` exclusion so Let's Encrypt (Certify The Web, HTTP-01) renewal is unaffected. Created and invoiced ticket #32376 (1h remote, prepaid).
Diagnosed and fixed the CyberSource credit-card payment outage ("Could not create SSL/TLS secure channel," 20 attempts / 0 success across all offices). Root cause: CyberSource (PNC processor) disabled TLS 1.0/1.1; the .NET payment library on Server 2019 negotiated old TLS. Applied `SchUseStrongCrypto=1` + `SystemDefaultTlsVersions=1` to both `.NETFramework\v4.0.30319` hives and recycled the `glaztech_new` app pool. Verified via the payments DB that card payments resumed (a real credit-card approval at 09:36 after the fix; eCheck path was never affected). Created and invoiced ticket #32377 (1.5h emergency remote, prepaid).
Performed a deep security assessment (read-only; no card numbers or passwords retrieved). Findings are severe: full PANs and CVV stored in plaintext, ~9,000+ plaintext passwords, SQL injection via a fake `quo()` escaper in payment pages, reflected XSS, debug/error disclosure, the production server doubling as a dev workstation, remote-access sprawl (incl. RealVNC 4.2.8), and a listener accepting TLS 1.0/1.1. Produced two reports and a high-priority client ticket (#32378) with a plain-language, point-by-point public writeup for Steve/Tom. Also traced why cards are stored (card-on-file invoice auto-pay via `gt_auto_process_2020.dll`) and confirmed the exposure is contained to the web app's databases (not Sage 100). Billed 1h remote to #32378 and left it open for remediation.
---
## Key Decisions
- **MailProtector digest = MailProtector-side issue.** Message trace proved Shannon gets no digest at all; the EOP SCL-bypass rule (created earlier) only helps users whose digests were being spam-filtered. No Exchange change made for Shannon.
- **Apex fix = add binding, not redirect.** Restored apex by adding host-header bindings to the existing site (apex serves same content as www) — fastest restore; cert already SAN-covered apex so no cert change.
- **HTTP→HTTPS redirect with ACME exclusion.** Used IIS URL Rewrite (installed) with a `/.well-known/acme-challenge/` negate condition so Certify/Let's Encrypt HTTP-01 renewal is not broken. 301 permanent.
- **Payment fix applied server-side first.** Set the .NET TLS keys + app-pool recycle (the standard fix). Verified success via the payments DB rather than waiting on Tom; the app-code `ServicePointManager.SecurityProtocol = Tls12` fallback was staged but not needed (DB confirmed live card approvals post-fix).
- **Security findings classified, never exfiltrated.** All card/password determinations used aggregate `CASE`/`COUNT` only — no PAN or password value was selected or displayed.
- **#32378 left OPEN (In Progress)** at Mike's direction for likely remediation work; billed but not resolved/closed.
- **Tokenization is the card remediation** — preserves the auto-pay feature while removing all stored PAN/CVV.
---
## Problems Encountered
- **`Get-MessageTraceV2` not exposed via `adminapi/InvokeCommand` initially** — investigator-exo token 401'd; the exchange-op token works and the cmdlet runs via InvokeCommand with a ≤10-day window constraint. The ExchangeOnlineManagement PowerShell module is NOT installed on GURU-5070 (used the REST InvokeCommand path instead).
- **PowerShell quoting/parse errors** in several RMM probes (escaped quotes inside `$(...)`, `foreach | Format-Table` empty-pipe, DBNull comparison). Resolved by collecting into variables, moving counts out of inline subexpressions, and DBNull-safe formatters.
- **Local apex test 404'd** because the binding is IP-scoped to 192.168.8.72 and `localhost` is 127.0.0.1 — verified externally instead (301/200/acme-404 all correct).
- **Syncro long-comment heredoc broke bash** (apostrophes inside `$(cat <<EOF)`). Resolved by writing the body to a file with a direct heredoc redirect, collapsing newlines to `<br>`, and posting via `jq -n --rawfile`.
---
## Configuration Changes
**On server WWW (192.168.8.72) — via GuruRMM:**
- IIS site `glaztech_new`: added bindings `http/192.168.8.72:80:glaztech.com` and `https/192.168.8.72:443:glaztech.com` (apex; reuses cert thumbprint `2684b127f039be5cb5adea6f3f616c111824e4dc`).
- `D:\web\glaztech_4\web.config`: inserted URL Rewrite rule "HTTP to HTTPS" (301, host-preserving, with `/.well-known/acme-challenge/` exclusion) as the first rule. Backup at `D:\web\glaztech_4\web.config.bak-20260603-090701`.
- Registry: `SchUseStrongCrypto=1` + `SystemDefaultTlsVersions=1` (DWORD) added to BOTH `HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319` and `HKLM\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v4.0.30319`; recycled app pool `glaztech_new`.
**Repo (ClaudeTools):**
- Created `clients/glaztech/reports/2026-06-03-pci-cardholder-data-finding.md`
- Created `clients/glaztech/reports/2026-06-03-website-security-assessment.md` (incl. attack-path + card data-flow sections)
**Syncro tickets:** #32376 (apex 404 + redirect), #32377 (payment TLS), #32378 (security assessment — left open).
---
## Credentials & Secrets
- **MailProtector / CloudFilter partner portal** — NO credentials in the vault. ACG manages it via manual partner-portal login. (Action: consider vaulting for future automation.)
- **MSP360 API** (unrelated, used 6/2): `msp-tools/msp360-api.sops.yaml`.
- **Exchange Operator app** (message trace): vault `msp-tools/computerguru-exchange-operator.sops.yaml`, app `b43e7342-5b4b-492f-890f-bb5a4f7f40e9`, tenant `82931e3c-de7a-4f74-87f7-fe714be1f160`; token via `get-token.sh <tenant> exchange-op`.
- **Glaztech SQL** (observed in site Web.config, NOT vaulted): server `192.168.8.62,3436`, login `tom` (password in `D:\web\glaztech_4\Web.config` connectionStrings), 15 per-office databases (`glaz_prod`, `glaz_prod_phx`, …, `glaz_prod_corp`) + `mas_gti` (Sage 100) + `qqest` (TimeForce).
---
## Infrastructure & Servers
- **WWW** — Windows Server 2019 Standard, build 10.0.17763.8755 (patched through May 2026). Local IP 192.168.8.72, public 65.113.52.88. IIS 10.0, .NET 4.8. Site `glaztech_new``D:\web\glaztech_4` (full VB.NET source on disk, not precompiled). LE cert `CN=www.glaztech.com`, SAN glaztech.com + www, expires 2026-08-19, via Certify The Web (HTTP-01). GuruRMM agent id `455a1bc7-1c29-42bc-b597-fa1e64f08eec`.
- **SQL** — `192.168.8.62,3436`, 15 office DBs + Sage 100 (`mas_gti`) + TimeForce (`qqest`).
- **Payment gateway** — CyberSource REST (`api.cybersource.com/pts/v2/payments`) as PNC merchant processor (card path); CyberSource SOAP toolkit for eCheck/ACH (`cybs.log`).
- **Email/spam** — MailProtector inbound filter `glaztech-com.inbound.emailservice.io`; digests from `noreply@azcomputerguru.com`.
---
## Commands & Outputs
- Apex 404: `curl -I https://glaztech.com` → 404 `Microsoft-HTTPAPI/2.0`; post-fix → 200; `/.well-known/acme-challenge/probe` over http → 404 plain (renewal safe).
- Payment TLS test (WWW → api.cybersource.com): TLS1.0/1.1 FAIL (SSPI), **TLS1.2 OK**; post-fix DB shows credit-card approval id 84455 at 09:36 (`web_payment_header`, glaz_prod_corp).
- Auth proc `get_web_accesslevel`: `where acct_no=@username and web_password=@passwd` (plaintext password compare).
- `quo()`: `Return "'" + stext + "'"` (no escaping → SQLi).
- `get_cc_data`: `select * from cc_file where acct_no=@acctno` (returns full PAN+CVV; IDOR-shaped).
- Card storage classification: `cc_file` ~780 rows total (plaintext PAN, 0 encrypted); `cc_file.cc_code` 50/54 CVV in Tucson; `cof_payments_header` Phoenix 14,496 rows / 11,794 plaintext PAN; `web_security` ~9,000+ plaintext passwords (corp 6,017 + tuc 3,012), 0 hash-like.
- Message trace (Shannon): 629 noreply digests in 10 days to ~60 recipients, 0 to shannon@glaztech.com.
---
## Pending / Incomplete Tasks
- **MailProtector:** provision/enable Shannon's Spam Summary in the MailProtector partner portal (manual). The earlier `gtimail@glaztech.com` daily digest failure (Priority-1 reject rule) is still open with Steve.
- **#32378 remediation (open ticket):** quick wins ready on Mike's go — purge stored CVV (`cc_file.cc_code`, backup-first, needs explicit sign-off), set `debug="false"` + `customErrors="On"`, remove RealVNC 4.2.8 + stale ScreenConnect, disable TLS 1.0/1.1. Then: hash passwords + stop emailing them, parameterize payment-page SQL (kill `quo()`), tokenize card-on-file (CyberSource vault), separate dev from prod, least-privilege DB.
- **Payment fix:** monitor that card successes continue; Tom code fallback (`SecurityProtocol = Tls12`) staged if needed.
- **Backups contain plaintext cards** — cleanup must address DB backups, not just live data.
---
## Reference Information
- Tickets: #32376 (id 112107224, apex 404+redirect, Resolved, 1h remote), #32377 (id 112109440, CyberSource TLS, Resolved, 1.5h emergency remote), #32378 (id 112111185, security assessment, **In Progress**, 1h remote).
- Glaztech Syncro customer id 143932; **prepaid block 26.5 → 22.25 hrs** today.
- Reports: `clients/glaztech/reports/2026-06-03-pci-cardholder-data-finding.md`, `…-website-security-assessment.md`.
- GuruRMM client id `d857708c-5713-4ee5-a314-679f86d2f9f9`; WWW agent `455a1bc7-1c29-42bc-b597-fa1e64f08eec`.
- TLS fix pattern (legacy .NET → modern gateway): `SchUseStrongCrypto=1` + `SystemDefaultTlsVersions=1` both `.NETFramework\v4.0.30319` hives + app-pool recycle.
- LE-safe HTTP→HTTPS redirect: URL Rewrite rule with `{REQUEST_URI}` `^/\.well-known/acme-challenge/` negate condition.