sync: auto-sync from GURU-5070 at 2026-06-04 19:08:11
Author: Mike Swanson Machine: GURU-5070 Timestamp: 2026-06-04 19:08:11
This commit is contained in:
Binary file not shown.
@@ -106,3 +106,41 @@ Ran a collaborative gap-analysis loop (Claude + Grok CLI, 4 Grok turns on sessio
|
||||
### Reference Information
|
||||
- Grok session `019e9351-ed1c-7bc3-b171-b4cf4b53745d`; SQL host `GTI-INV-SQL` `192.168.8.62,3436` (instance `GTISQL`).
|
||||
- Coord todos `6d15fc88-db4f-4a35-a76a-a5a6a9f50795`, `aebaf751-d778-423f-a84b-314fbb294f30`.
|
||||
|
||||
---
|
||||
|
||||
## Update: 19:07 PT — Glaztech infra remediation blitz via RMM (dev-tool removal, web hardening, domain time fix, ACL, sa)
|
||||
|
||||
### Session Summary
|
||||
Executed a large batch of Glaztech remediation through GuruRMM. WWW was already enrolled; mid-session Mike enrolled the **DCs + SQL server**, which unlocked the domain/SQL-side work. All actions were RMM-driven, verified, and caused **no outages**. Several changes were scheduled (off-hours) with backup + health-check + auto-rollback. One acute item (msdb plaintext domain-admin removal) is **paused awaiting method approval**.
|
||||
|
||||
### Completed (all verified, no outage)
|
||||
- **Dev tooling removed from WWW (H1):** VS 2015 + 2022 (~15.6 GB reclaimed, via bootstrapper `/uninstall /quiet /norestart`), IIS Express, Notepad++, OpenSSL, RealDownloader (+ scheduled task). Archived D7x (`D:\d7`, `D:\d7x Resources`), `D:\3rd Party Tools`, `D:\Scripts`, `D:\bin` (CyberSource SDK sample), and web-root `Old_code`/`Old_bin`/26 `.pdb` to **`D:\_removed_devtools_2026-06-04\`** (reversible). One-time reboot finalized the VS `PendingFileRenameOperations` (cleared).
|
||||
- **WWW Web.config hardening** (scheduled task `ACG-WebConfigHarden-20260604` @ box-17:05, applied + survived the 17:15 reboot): `debug=false`; security headers `X-Content-Type-Options: nosniff`, `X-Frame-Options: SAMEORIGIN`, `Referrer-Policy: strict-origin-when-cross-origin`, `Strict-Transport-Security: max-age=31536000`; `httpCookies httpOnlyCookies=true requireSSL=true`; **CORS scoped to `<location path="emails">` (Origin:* Methods:GET)** and the site-wide wildcard CORS removed. Backup `D:\web\glaztech_4\Web.config.bak-20260604-170500`. Live headers verified.
|
||||
- **Domain time fixed end-to-end:** PDC **GTI-INV-DC** was syncing from the Hyper-V host (VM IC provider) and drifting → re-pointed to **external NTP (pool.ntp.org)**, `VMICTimeProvider` disabled, marked reliable. **GTI-INV-DC1** → follows PDC. **WWW** (was `Local CMOS Clock`/free-running, ~8 min slow) re-registered (`w32tm /unregister`+`/register`) → PDC, clock **stepped +8 min**. **GTI-INV-SQL** → DC1. All four converged within ~3 s. Kerberos-skew resolved.
|
||||
- **WWW `Everyone:(R)` ACL (E1):** removed from **`Web.config` + `bin`** (granted `IIS_IUSRS` + `IIS APPPOOL\glaztech_new` RX first; site stayed HTTP 301). Public static content (`emails/`,`images/`) left as a low-priority slower sweep.
|
||||
- **GTI-INV-SQL: built-in `sa` disabled** (re-check showed **0 real user sessions**; the 29 "active" sessions were all `is_user_process=0` system sessions). Done via WWW's app `tom` connection (SYSTEM-on-SQL is not sysadmin).
|
||||
|
||||
### Key Findings
|
||||
- **WWW clock** was never syncing (free-running) — ~6–8 min slow; surfaced when Mike noticed it. **PDC** itself was VM-host-timed, not NTP.
|
||||
- **Forest = `glaztech.local` (root) + `glaztech.com` (child).** **NS4.glaztech.local holds the Schema-master FSMO but is a DEAD server** (per Mike) → orphaned FSMO; external NTP on GTI-INV-DC is correct (can't chain to dead root).
|
||||
- **CORS:** the wildcard `Access-Control-Allow-Origin: *` was only used by cross-origin loads of `/emails/` assets (IIS logs: 188 OPTIONS, 181 → `/emails/`; none to the API/payment surface) → scoped to `/emails`.
|
||||
- **msdb cleartext cred:** **11 TSQL backup-copy job steps** embed `net use \\192.168.8.52|.212\sql_backup\... /user:glaztech\administrator <pw> /persistent:yes` + a `copy`. They run as the SQL **engine** service account (machine acct, no share access) → can't just blank the creds. **0 existing SQL credentials/proxies; SQL Agent service account = `Administrator@glaztech.com` (domain admin).**
|
||||
|
||||
### Key Decisions
|
||||
- **Web.config / ACL health checks must hit the real binding `http://192.168.8.72/` (Host: www.glaztech.com), NOT `127.0.0.1`** — the site binds to the LAN IP only. Caught + fixed the scheduled apply's health check at box-17:03, ~90 s before the 17:05 run (the 127.0.0.1 check would have false-rolled-back the change). **Reusable rule for future WWW scripts.**
|
||||
- ACL fix scoped to `Web.config`+`bin` (the secrets/assemblies) instead of a slow full-tree `/T` (static content is public anyway).
|
||||
- All scheduled/unattended changes built with backup + post-change health-check + auto-rollback; reachability-gated for the PDC NTP change (rollback to host time if NTP unreachable — it was reachable).
|
||||
|
||||
### PENDING — pick up next session
|
||||
- **msdb plaintext removal — AWAITING GO on method.** Recommended: **SQL Credential + Agent CmdExec proxy** (encrypt the pw in `sys.credentials`, convert the 11 steps to CmdExec-under-proxy, drop inline creds; decoupled from Agent privilege; `ALTER CREDENTIAL` after rotation). Alt: `cmdkey` + strip inline. Test-first + snapshot originals to admin-only file (deleted after) + verify a copy works.
|
||||
- **Rotate `glaztech\administrator`** — Mike coordinating with **Steve** (deferred). Identify all consumers first.
|
||||
- Gated/heavier: **disable `xp_cmdshell`** (blocked until the 11 backup-copy steps are reworked — they depend on it); **disable TLS 1.0/1.1 on WWW** (needs reboot); **full web-root `Everyone` sweep** (low pri); **seize/clean Schema-master FSMO off dead NS4**; de-privilege the SQL Agent account.
|
||||
- WWW one-time scheduled tasks `ACG-WebConfigHarden-20260604` + `ACG-Reboot-VSCleanup-20260604` both fired Result=0 (can be deleted or left).
|
||||
|
||||
### Infrastructure / Reference
|
||||
- **Glaztech RMM agents** (client "Glaztech Industries"): `WWW` 455a1bc7 (site TUS-Tucson), `GTI-INV-DC` 0337e973 (**PDC**, INV-Involta), `GTI-INV-DC1` ffcaafac, `GTI-INV-SQL` 869e56b4. (NOT Glaztech: `SAGE-SQL`=Dataforth, `ACG-DC16`=ACG, `VWP-DC1`=VWP.)
|
||||
- Domain `glaztech.com` (member servers); forest root `glaztech.local` (NS4, dead). Backup file servers `\\192.168.8.52\sql_backup`, `\\192.168.8.212\sql_backup`. SQL instance `GTISQL` @ `192.168.8.62,3436`.
|
||||
- On-WWW logs: `C:\temp\{vs_uninstall, devtools_groupB, groupC, acl_fix, acl_fix2, sa_via_tom, webconfig_apply}.log`; on DCs/SQL: `C:\temp\timefix_*.log`.
|
||||
- Local scripts (this machine): `C:\Users\guru\AppData\Local\Temp\grok_glaztech\*.ps1`.
|
||||
- Coord locks held: `clients/glaztech:glaztech/domain-time` (61cd25f2), `clients/glaztech:WWW/devtools-removal` (c4226bac).
|
||||
|
||||
Reference in New Issue
Block a user