wiki: seed Instrumental Music Center + Valley Wide Plastering articles
instrumental-music-center.md — AIMsi POS on SQL Server 2019 (Standard under misleading SQLEXPRESS instance name); phantom DC ServerIMC causing slow logons; GuruRMM enrolled (IMC1 fa99e913); OpenVPN subnet-overlap hazard; $175/hr prepaid, 12.5 hrs remaining; SQL max server memory fix approved but unverified applied. valleywide.md — Valley Wide Plastering; HP DL360 Gen10 VM host + XenServer; VB6/Access 97 app modernization (130 tables, 791 Crystal Reports, certified payroll); RDWeb brute-force incident (contained); 11 Yealink phones pending; iLO requires paramiko (legacy ssh-rsa); $175/hr prepaid, 10 hrs remaining. wiki/index.md — both clients added to Clients table and Cross-Reference. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
325
wiki/clients/instrumental-music-center.md
Normal file
325
wiki/clients/instrumental-music-center.md
Normal file
@@ -0,0 +1,325 @@
|
||||
---
|
||||
type: client
|
||||
name: instrumental-music-center
|
||||
display_name: Instrumental Music Center
|
||||
last_compiled: 2026-05-24
|
||||
compiled_by: DESKTOP-0O8A1RL/claude-main
|
||||
sources:
|
||||
- clients/instrumental-music-center/README.md
|
||||
- clients/instrumental-music-center/PROJECT_STATE.md
|
||||
- clients/instrumental-music-center/docs/overview.md
|
||||
- clients/instrumental-music-center/docs/billing-log.md
|
||||
- clients/instrumental-music-center/docs/2026-04-13-ticket-notes.md
|
||||
- clients/instrumental-music-center/docs/network/topology.md
|
||||
- clients/instrumental-music-center/docs/network/vlans.md
|
||||
- clients/instrumental-music-center/docs/network/firewall.md
|
||||
- clients/instrumental-music-center/docs/network/dhcp.md
|
||||
- clients/instrumental-music-center/docs/network/dns.md
|
||||
- clients/instrumental-music-center/docs/cloud/m365.md
|
||||
- clients/instrumental-music-center/docs/cloud/azure.md
|
||||
- clients/instrumental-music-center/docs/rmm/rmm.md
|
||||
- clients/instrumental-music-center/docs/security/antivirus.md
|
||||
- clients/instrumental-music-center/docs/security/backup.md
|
||||
- clients/instrumental-music-center/docs/issues/log.md
|
||||
- clients/instrumental-music-center/docs/servers/server_template.md
|
||||
- clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md
|
||||
- clients/instrumental-music-center/session-logs/2026-04-28-howard-manda-laptop-provision.md
|
||||
- clients/instrumental-music-center/session-logs/2026-05-04-station2-printer-and-manda-vpn.md
|
||||
- clients/instrumental-music-center/session-logs/2026-05-05-howard-aim-connection-broken-investigation.md
|
||||
- clients/instrumental-music-center/session-logs/2026-05-06-howard-imc1-aim-instance-correction.md
|
||||
- clients/instrumental-music-center/decisions/2026-05-07-mike-memory-allocation-approval.md
|
||||
backlinks:
|
||||
- projects/gururmm
|
||||
---
|
||||
|
||||
# Instrumental Music Center
|
||||
|
||||
Music retail and instrument repair shop running AIMsi point-of-sale software on-prem. Single-site as far as documented. Located at 7063 E Speedway Blvd, Tucson AZ 85710. ACG provides managed break-fix / prepaid-block support; primary focus is on the AIMsi SQL server (IMC1) and workstation fleet.
|
||||
|
||||
---
|
||||
|
||||
## Profile
|
||||
|
||||
- **Contract type:** Prepaid hour block
|
||||
- **Billing rate:** $175/hr all labor
|
||||
- **Hours remaining:** 12.5 hrs as of 2026-04-28 (after debiting 1.5 hrs for Syncro #32218). Always live-check before billing.
|
||||
- **Syncro customer ID:** 7088508
|
||||
- **Key contacts:**
|
||||
- **Leslie Stirm** — primary contact; leslie@imc-az.com; Syncro contact_id 731730
|
||||
- **Manda** — General Manager (new, replaced Michael Santander as of ~2026-04-28). Full name unconfirmed in AD. [unverified]
|
||||
- **Michael Santander** — former GM; domain account already deactivated.
|
||||
- **Primary domain:** imc.local (on-prem AD)
|
||||
- **Location:** Speedway (7063 E Speedway Blvd, Tucson AZ 85710) — additional locations TBD; only Speedway is documented.
|
||||
- **Critical software:** AIMsi by Tri-Tech (https://www.tritechretail.com/topic/aim) — retail POS and inventory management.
|
||||
|
||||
---
|
||||
|
||||
## Infrastructure
|
||||
|
||||
### Servers & Services
|
||||
|
||||
| Host | IP | Role | OS | Notes |
|
||||
|---|---|---|---|---|
|
||||
| IMC1 | 192.168.0.2 | DC (imc.local), DNS, File Server, AIMsi SQL host, RDS host | Windows Server 2016 Standard (build 14393.7426) | Dell R720, 4 physical cores, 32 GB RAM. GuruRMM agent: `fa99e913-1027-4e33-a928-7695e31068e7` |
|
||||
| ServerIMC | 192.168.0.63 | Phantom / broken DC | Windows Server 2016 Essentials [unverified] | **[WARNING] Registered as DC in AD DNS (A + SRV records for `_ldap._tcp.dc._msdcs.imc.local` and `_kerberos._tcp.imc.local`). Responds to ICMP but TCP/389 (LDAP) and TCP/88 (Kerberos) refuse connections. DC locator round-robins — clients that pick ServerIMC time out. Root cause of intermittent slow logons, GPO failures, and 2026-04-22 remote domain-join failure. Needs `ntdsutil` metadata cleanup (if demoted ghost) or AD service repair.** |
|
||||
| IMC2 | — | Unknown (stale) | Windows Server 2016 Essentials | Last logon 2023 — likely decommissioned. Clean up AD computer object. |
|
||||
| IMC-VM | — | Unknown (dead) | Windows Server 2016 Standard | Last logon 2021 — dead. Clean up AD computer object. |
|
||||
| Station 1 | 192.168.0.50 | POS workstation | Windows [unverified] | Hostname `IMC-STATION1`. Primary workstation for AIM "connection broken" incidents. |
|
||||
|
||||
#### IMC1 SQL Instances (CRITICAL — read carefully)
|
||||
|
||||
**[WARNING] The production AIM database is on `IMC1\SQLEXPRESS`, NOT `IMC1\AIMSQL`. The instance name is actively misleading — someone installed SQL Server 2019 Standard under the default `SQLEXPRESS` instance name and never renamed it. This burned a full day of triage. Always verify SQL roles by active connections (`sys.dm_exec_sessions`) — never by instance name.**
|
||||
|
||||
| Instance | Port | Edition (actual) | Role | Production DB | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| `IMC1\SQLEXPRESS` | TCP 61151 | **SQL Server 2019 Standard** (misleading name!) | **PRODUCTION** | `IMCAIM` (created 2023-08-21) | Service account `IMC\AIM`. ~9 store workstations + 22 server-local AIM sessions. **Do not stop, do not uninstall.** ERRORLOG at `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Log\`. No `max server memory` cap (default unlimited). |
|
||||
| `IMC1\AIMSQL` | TCP 63116 (dynamic) | SQL Server 2019 Express GDR 15.0.2165.1 | **Orphan** (consolidation candidate) | None active | Service account `IMC\IMC1$`. Zero established TCP connections. Holds only 2023-era conversion-test DBs (`AIM`, `IMC`, `TestConv61223`). No active backup chain landing here. Shutdown + uninstall approved by Mike pending `.mdf` backup confirmation. |
|
||||
| `IMC1\MICROSOFT##WID` | — | Windows Internal Database | WSUS / AD RMS | — | WSUS confirmed NOT in use at IMC. AD RMS status unverified. If AD RMS also unused, instance can be stopped to free ~300 MB. **Canary for memory pressure** — Event 17890 paging events fire here first when the host is memory-squeezed. |
|
||||
|
||||
**Workstations connected to production `IMC1\SQLEXPRESS` (verified 2026-05-06):**
|
||||
|
||||
| Hostname | IP |
|
||||
|---|---|
|
||||
| IMC-MINI | 192.168.0.72 |
|
||||
| IMC-SVCSTR | 192.168.0.55 |
|
||||
| IMC-LESSONS | 192.168.0.62 |
|
||||
| IMC-STATION2 | 192.168.0.66 |
|
||||
| IMC-L1-STATION9 | 192.168.0.41 |
|
||||
| DESKTOP-44L80C0 | 192.168.0.46 |
|
||||
| DESKTOP-MR3ALTK | 192.168.0.59 |
|
||||
| REPAIRADMIN | 192.168.0.48 |
|
||||
| C2B | 192.168.0.4 |
|
||||
| IMC-STATION1 | 192.168.0.50 |
|
||||
|
||||
All sessions authenticate as `AIMUser1` via `.Net SqlClient Data Provider`.
|
||||
|
||||
#### IMC1 Disk Layout
|
||||
|
||||
| Drive | Purpose | Notes |
|
||||
|---|---|---|
|
||||
| C: | OS, IIS, system DBs | 419 GB volume; ~278 GB used after 2026-04-12 cleanup (~66%); was 77% full before. Monitor. |
|
||||
| E: | SQL backups + installers + Server 2016 media | `E:\W2016\sources\install.wim` is RTM 14393.0. SQL backups at `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\` |
|
||||
| F: | Windows Image Backups | — |
|
||||
| S: | Dedicated SSD (Samsung 850 PRO 256 GB) — AIMsi SQL user DBs | User DBs at `S:\SQL\Data\`. AIM client share `\\IMC1\AIM` → `S:\AIM`. System DBs remain on C:. |
|
||||
|
||||
### Email & Identity
|
||||
|
||||
- **Mail:** IMC uses a **mixed Google / Microsoft identity model** — different users are on different platforms. Manda is on the M365 side. [full tenant details unverified]
|
||||
- **M365 tenant details:** Not fully documented. Manda's Outlook was configured against an existing M365 mailbox.
|
||||
- **On-prem AD domain:** `imc.local`
|
||||
- **MFA status:** [unverified]
|
||||
- **DNS:** IMC1 (192.168.0.2) is the authoritative DNS server for imc.local. ServerIMC (192.168.0.63) has ghost A + SRV records as a DC — these are the direct cause of client authentication failures and need cleanup.
|
||||
|
||||
### Network
|
||||
|
||||
- **LAN subnet:** 192.168.0.0/24
|
||||
- **VPN:** OpenVPN (.ovpn profile). **[WARNING] 192.168.0.0/24 subnet overlap hazard:** if technician's home/office LAN is also 192.168.0.0/24 (Howard's home is), OpenVPN routes win for reaching IMC1 but Windows multi-homed DNS races between the two interfaces. DNS negative caching causes domain join / locator failures. **If remote LAN overlaps IMC's subnet, go onsite for domain joins.** Also: disconnect Tailscale before connecting to IMC OpenVPN — Tailscale's `pfsense-2` subnet router advertises 192.168.0.0/24 with lower metric than the VPN, making IMC1 unreachable.
|
||||
- **Firewall:** [unverified — not documented]
|
||||
- **ISP:** [unverified]
|
||||
- **SMB:** SMB1 still enabled on IMC1 — disable as security hygiene when opportunity permits.
|
||||
- **SMB signing:** `RequireSecuritySignature = True` on server — adds auth overhead.
|
||||
|
||||
---
|
||||
|
||||
## GuruRMM Enrollment
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| GuruRMM client | Instrumental Music Center |
|
||||
| GuruRMM client ID | `213b62a8-30f4-41dd-9bb3-549341104416` |
|
||||
| GuruRMM client code | `IMC` |
|
||||
| Site | IMCMain |
|
||||
| Site ID | `2c5b65ad-2d5e-47b3-b12b-632e35e08ff6` |
|
||||
| Site code | `INNER-BRIDGE-8354` |
|
||||
| Site enrollment key | vault: `clients/imc/gururmm-site-main.sops.yaml` |
|
||||
| First enrolled agent | IMC1 (`fa99e913-1027-4e33-a928-7695e31068e7`) |
|
||||
|
||||
IMC was enrolled in GuruRMM on 2026-05-05 (Howard, prompted by AIM connection-broken investigation). IMC1 agent was installed by Mike via ScreenConnect. Only IMC1 is enrolled as of last session — workstations not yet enrolled.
|
||||
|
||||
**Note:** When SSH from Howard-Home is blocked by the 192.168.0.0/24 route collision, GuruRMM remote commands are the fallback for running diagnostics on IMC1.
|
||||
|
||||
---
|
||||
|
||||
## Access
|
||||
|
||||
- **SSH:** `ssh IMC\guru@192.168.0.2` — ed25519 key auth; PowerShell is the default shell. Authorized keys: `C:\ProgramData\ssh\administrators_authorized_keys` (inheritance off, Administrators + SYSTEM full control).
|
||||
- **VPN:** OpenVPN (.ovpn profile). Disconnect Tailscale first. If home/office LAN is 192.168.0.0/24, remote domain operations will fail — go onsite instead.
|
||||
- **Domain admin:** `IMC\guru` — also SQL sysadmin on both SQLEXPRESS and AIMSQL (added via single-user recovery 2026-04-12).
|
||||
- **GuruRMM:** IMC1 agent `fa99e913-1027-4e33-a928-7695e31068e7` — use for remote commands when SSH is blocked.
|
||||
- **Vault paths:**
|
||||
- IMC1 credentials (domain admin, SSH): `clients/imc/imc1.sops.yaml`
|
||||
- GuruRMM site enrollment key: `clients/imc/gururmm-site-main.sops.yaml`
|
||||
|
||||
**[WARNING] `sa` account on AIMSQL:** exists and enabled; password unknown. One candidate was tried and failed on 2026-04-12 — no lockout triggered (no lockout policy). If needed for AIMSQL consolidation, use single-user recovery mode (same process used 2026-04-12).
|
||||
|
||||
---
|
||||
|
||||
## AIMsi / Tri-Tech Critical Notes
|
||||
|
||||
**Per-machine workstation number (`USER#`) is mandatory.** AIMsi requires a user environment variable `USER#` (older Tri-Tech convention, still in use at IMC) set on each machine. This is the per-machine workstation identifier for POS polling and licensing.
|
||||
|
||||
- **NEVER wipe or reimage a machine without recording its `USER#` first.**
|
||||
- **When deploying a new machine, assign its `USER#` per Leslie** — she tracks the allocation.
|
||||
- Tri-Tech docs: https://www.tritechretail.com/topic/aim
|
||||
|
||||
**Known `USER#` assignments:**
|
||||
|
||||
| Machine | Hostname | USER# | Notes |
|
||||
|---|---|---|---|
|
||||
| Manda (GM) laptop | DESKTOP-KRHQ5TS | 4 | Assigned per Leslie, 2026-04-28 |
|
||||
| Other workstations | Various | TBD | Not yet fully documented |
|
||||
|
||||
---
|
||||
|
||||
## Backups
|
||||
|
||||
- **Local SQL backups:** Nightly at 22:00 to `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\IMCAIM_*.bak`
|
||||
- **Retention script:** `C:\Scripts\Clean-AimsiBackups.ps1` — GFS policy: 14 dailies + 1st-of-month; 3-newest safety override; logs to `C:\Scripts\Logs\aimsi-retention-YYYYMM.log`
|
||||
- **Retention task:** `IMC AIMsi Backup Retention` — daily 23:30, SYSTEM, 1-hour limit
|
||||
- **Off-site:** Cloudberry / MSP360 at `C:\ProgramData\Online Backup\`. Cloudberry chain confirmed intact before 2026-04-12 deletion run.
|
||||
- SQLEXPRESS backup also confirmed landing at `C:\ProgramData\Online Backup\MSSQL\IMC1_SQLEXPRESS\`
|
||||
- **Windows Image Backup:** on F:
|
||||
- **AIMSQL orphan:** no backup chain. Locate and back up `AIM.mdf`, `IMC.mdf`, `TestConv61223.mdf` and their `.ldf` siblings before any consolidation — files were not found in expected path under `MSSQL15.AIMSQL\MSSQL\DATA` or `S:\*AIMSQL*` during 2026-05-06 search.
|
||||
|
||||
---
|
||||
|
||||
## Patterns & Known Issues
|
||||
|
||||
### [WARNING] Phantom DC `ServerIMC` — Active Authentication Degrader
|
||||
|
||||
`ServerIMC` (192.168.0.63) is registered in DNS as a domain controller (A record + SRV records for `_ldap._tcp.dc._msdcs.imc.local` and `_kerberos._tcp.imc.local`) alongside IMC1. It responds to ICMP ping but TCP/389 and TCP/88 refuse connections. The DC locator round-robins between IMC1 and ServerIMC, timing out ~50% of the time.
|
||||
|
||||
**Effect:** Intermittent slow logons, GPO failures, and broken remote domain joins for every domain client at IMC. Was the confirmed root cause of the 2026-04-22 failed remote join of `DESKTOP-KRHQ5TS`.
|
||||
|
||||
**Action needed:** Open a ticket. Either:
|
||||
1. Repair AD services if `ServerIMC` is a real machine with broken services, or
|
||||
2. Run `ntdsutil` metadata cleanup if it is a ghost from a previously demoted DC.
|
||||
|
||||
This was first flagged as "unclear" on 2026-04-13, promoted to confirmed issue 2026-04-28. No ticket has been opened as of 2026-05-06.
|
||||
|
||||
### AIM "Connection Broken" — Memory Pressure on IMC1
|
||||
|
||||
**Symptom:** `Telerik.OpenAccess.RT.sql.SQLException: Connection has been closed / The connection is broken and recovery is not possible` — user-facing AIM crash. First seen 2026-05-05 on Station 1 (IMC-STATION1, 192.168.0.50), recurred 2026-05-06 ~12:14 PM.
|
||||
|
||||
**Root cause:** IMC1 is hosting DC services + 6 concurrent RDP users + AIMsi Webservice/Runtime + three SQL instances + QuickBooks Enterprise on 32 GB. Under memory pressure, Windows trims SQL working sets (visible as WID Event 17890 paging events — the canary). The trim reaps idle Telerik OpenAccess TCP pool slots. Telerik has no transient-fault retry, so the next query against a dead pool handle throws the raw stack trace.
|
||||
|
||||
**SQLEXPRESS has no `max server memory` cap** (default 2,147,483,647 MB). Working set observed at 6.86 GB.
|
||||
|
||||
**Approved fix (Mike, 2026-05-07):** Cap `max server memory` on each instance:
|
||||
- `SQLEXPRESS`: 12,288 MB (12 GB)
|
||||
- `MSSQL$MICROSOFT##WID`: 512 MB
|
||||
- `MSSQL$AIMSQL`: 256 MB (or consolidate it)
|
||||
|
||||
**Status as of 2026-05-06:** Howard is awaiting go-ahead for implementation. Mike approved on 2026-05-07. **Confirm whether Howard has applied the caps — this is the immediate recurrence prevention.** [unverified post-2026-05-07]
|
||||
|
||||
### [WARNING] SQL Instance Name Trap
|
||||
|
||||
**`IMC1\SQLEXPRESS` is SQL Server 2019 Standard Edition** — someone installed Standard under the default `SQLEXPRESS` instance name and never renamed it. `SERVERPROPERTY('Edition')` is the only way to confirm this. The instance name actively misleads.
|
||||
|
||||
**Never assume an instance is idle, orphan, or Express based on name.** Always verify by:
|
||||
1. `SERVERPROPERTY('Edition')` for edition
|
||||
2. `sys.dm_exec_sessions` for active user sessions
|
||||
3. `Get-NetTCPConnection -OwningProcess` for established TCP connections
|
||||
|
||||
This trap caused a wrong-instance restart task to be deployed (2026-05-05) that had zero effect on the user-facing problem and was unregistered the next day (2026-05-06). See `.claude/memory/feedback_sql_instance_role_by_connection.md`.
|
||||
|
||||
### Component Store Corruption on IMC1 (RDS Removal Blocked)
|
||||
|
||||
`COMPONENTS` registry hive is ~168 MB (normal 30-50 MB), causing `0x80073701 ERROR_SXS_ASSEMBLY_MISSING` on any role removal or CU apply-on-boot. ETW manifest for provider GUID `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}` is malformed — causes `CBS_E_INSTALLERS_FAILED` → full rollback even when CU staging succeeds.
|
||||
|
||||
**Effect:** Blocks RDS role removal, which was the original reason for the 2026-04-12 engagement. Also means CU KB5075999 cannot be applied cleanly.
|
||||
|
||||
**Server is otherwise healthy** — AIMsi production is running. This is a structural impediment to the Server 2019 migration. Three paths considered (see History Highlights).
|
||||
|
||||
### Remote Domain Join Over OpenVPN — Don't Do It
|
||||
|
||||
If the technician's local LAN subnet overlaps IMC's 192.168.0.0/24, remote domain joins over OpenVPN will fail reliably:
|
||||
- OpenVPN pushed routes win for TCP, but Windows multi-homed DNS races between LAN DNS and VPN DNS (both respond to `imc.local` queries; LAN returns NXDOMAIN faster; Windows caches the negative answer).
|
||||
- Even with NRPT rules, hosts file entries, `-Server <IP>` on Add-Computer, and `nltest /dsgetdc /force` — the combination of subnet overlap + phantom DC (ServerIMC) beat all client-side workarounds.
|
||||
|
||||
**Rule:** For IMC domain operations where local subnet overlap exists, go onsite.
|
||||
|
||||
### Mixed Email Identity (Google + M365)
|
||||
|
||||
IMC users are split between Google Workspace and Microsoft 365 — different users on different platforms. When configuring a new user, confirm with Leslie which platform their mailbox lives on before setting up Outlook vs. Gmail.
|
||||
|
||||
### Stale AD Objects
|
||||
|
||||
| Object | Last Logon | Status | Action |
|
||||
|---|---|---|---|
|
||||
| IMC2 (computer) | 2023 | Likely decommissioned | Clean up AD object |
|
||||
| IMC-VM (computer) | 2021 | Dead | Clean up AD object |
|
||||
| ServerIMC (DC) | Active (ICMP) | Phantom/broken DC | ntdsutil metadata cleanup or repair |
|
||||
|
||||
### GPO Noise
|
||||
|
||||
- **DistributedCOM 10016** fires every 5 minutes — RuntimeBroker permission noise. Cosmetic.
|
||||
- **Group Policy event 103** fires every 5 minutes — "removal of the assignment of application Syncro from policy Management SW failed". Stale GPO object. Cleanup separately.
|
||||
|
||||
### Server 2016 EOL
|
||||
|
||||
Extended support ends **2027-01-12**. Migration window is finite. The memory pressure / AIM reliability incident is additional evidence to push the migration timeline. Mike wants to scope cost/timeline at next ACG strategy call.
|
||||
|
||||
---
|
||||
|
||||
## Active Work
|
||||
|
||||
As of 2026-05-07 (last decision recorded):
|
||||
|
||||
1. **[IMMEDIATE] Apply `max server memory` caps on IMC1 SQL instances** — Mike approved 2026-05-07. Howard to implement: SQLEXPRESS 12 GB, WID 512 MB, AIMSQL 256 MB. Reversible (1-second config change, no service restart). Until applied, AIM connection-broken errors will continue recurring. [unverified — confirm applied]
|
||||
|
||||
2. **[HIGH] Open ticket for ServerIMC phantom DC investigation** — SRV/A records in DNS claim it's a DC; LDAP/Kerberos refuse connections. Degrades authentication for all domain users. No ticket opened as of 2026-05-06.
|
||||
|
||||
3. **[MEDIUM] AIMSQL orphan consolidation** — Mike approved (2026-05-07). Pending:
|
||||
- Locate `AIM.mdf`, `IMC.mdf`, `TestConv61223.mdf` and `.ldf` siblings (not in expected path)
|
||||
- Back up 2023-era DBs before shutdown
|
||||
- Verify no applications reference `IMC1\AIMSQL` (TCP 63116)
|
||||
- Stop and uninstall `MSSQL$AIMSQL`
|
||||
|
||||
4. **[MEDIUM] WID instance decision** — Verify AD RMS usage. WSUS confirmed unused. If AD RMS also unused, stop WID to free ~300 MB headroom. Mike awaiting Howard's verification before authorizing stop.
|
||||
|
||||
5. **[LOWER] Server 2019 migration scoping** — Three paths (component store repair + in-place; in-place without repair; clean build). Clean build is Mike's recommendation. Scope cost/timeline at next ACG strategy call before 2027-01-12 EOL.
|
||||
|
||||
6. **[LOWER] Documentation cleanup:**
|
||||
- Update workstation table in `docs/overview.md` with `DESKTOP-KRHQ5TS` / Manda / AIM USER#=4
|
||||
- Confirm Manda's full name in AD
|
||||
- Disable SMB1 on IMC1 (`Set-SmbServerConfiguration -EnableSMB1Protocol $false`)
|
||||
- Drop `TestConv61223` DB on AIMSQL (leftover 2023 migration test) — safe per enumeration, but back up `.mdf` first
|
||||
- Clean up stale AD computer objects `IMC2`, `IMC-VM`
|
||||
|
||||
---
|
||||
|
||||
## History Highlights
|
||||
|
||||
| Date | By | Event |
|
||||
|---|---|---|
|
||||
| ~2026-Q1 | Mike/Howard | Early engagement: 3 new workstations provisioned at Speedway (hostnames, AIM USER#s TBD in billing log) |
|
||||
| 2026-04-11/12/13 | Mike | IMC1 maintenance: RDS removal blocked (component store corruption 0x80073701), SSH installed, 716 GB freed on E: (backup cleanup), GFS retention automated, AIMsi DBs moved C:→S: SSD |
|
||||
| 2026-04-22 | Howard | Attempted remote domain-join of `DESKTOP-KRHQ5TS` over VPN — abandoned after subnet overlap + phantom DC defeated all workarounds |
|
||||
| 2026-04-28 | Howard | Onsite: `DESKTOP-KRHQ5TS` joined to imc.local, Manda (new GM) AD account created, Outlook/M365 configured, Office activated, AIMsi USER#=4 set. Ticket #32218, 1.5 hrs, prepay 14.0→12.5 hrs. ServerIMC confirmed as active authentication degrader. |
|
||||
| 2026-05-04 | Howard | Onsite (0.5 hrs): Station 2 receipt printer reconnected (re-added from \\imc1); VPN installed on Manda's machine. Ticket #32247. |
|
||||
| 2026-05-05 | Howard | AIM "connection broken" investigation. GuruRMM IMC client/site provisioned, IMC1 enrolled. Diagnosed memory pressure; scheduled AIMSQL restart for 02:30 (wrong instance — superseded next day). |
|
||||
| 2026-05-06 | Howard | Station 1 recurrence 12:14 PM. Full instance enumeration revealed SQLEXPRESS = production Standard (not AIMSQL). Wrong-instance restart task unregistered. Corrected diagnosis in session logs and PROJECT_STATE. Feedback memory created. |
|
||||
| 2026-05-07 | Mike | Decision: approved memory caps (SQLEXPRESS 12 GB, WID 512 MB, AIMSQL 256 MB), AIMSQL consolidation pending backup, Server 2016 migration timeline acknowledged, WSUS confirmed unused. |
|
||||
|
||||
---
|
||||
|
||||
## Compilation Notes
|
||||
|
||||
Source material: 5 session logs (2026-04-12 through 2026-05-06) + 1 decision file (2026-05-07) + README + PROJECT_STATE + 10 docs files (most docs/* are blank templates with no client-specific data filled in — network/firewall/vlans/VLAN/DHCP/DNS/RMM/AV/backup/issues docs are all empty templates).
|
||||
|
||||
Many structured docs (`docs/network/`, `docs/security/`, `docs/cloud/`) are empty templates. The authoritative information sources are `README.md`, `PROJECT_STATE.md`, and the session logs.
|
||||
|
||||
**Unverified items flagged:**
|
||||
- Whether Howard applied `max server memory` caps after Mike's 2026-05-07 approval
|
||||
- ServerIMC ticket status — ticket was recommended but not confirmed opened
|
||||
- Manda's full name in AD
|
||||
- M365 tenant details (tenant domain, license type, MFA policy)
|
||||
- WID instance AD RMS usage
|
||||
- AIMSQL `.mdf` file locations
|
||||
- Full workstation fleet AIM USER# assignments
|
||||
- ISP, firewall hardware, VLAN/network topology
|
||||
|
||||
## Backlinks
|
||||
|
||||
- [[projects/gururmm]] — IMC1 enrolled as agent `fa99e913-1027-4e33-a928-7695e31068e7`; site IMCMain
|
||||
271
wiki/clients/valleywide.md
Normal file
271
wiki/clients/valleywide.md
Normal file
@@ -0,0 +1,271 @@
|
||||
---
|
||||
type: client
|
||||
name: valleywide
|
||||
display_name: Valley Wide Plastering
|
||||
last_compiled: 2026-05-24
|
||||
compiled_by: DESKTOP-0O8A1RL/claude-main
|
||||
sources:
|
||||
- clients/valleywide/README.md
|
||||
- clients/valleywide/PROJECT_STATE.md
|
||||
- clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md
|
||||
- clients/valleywide/session-logs/2026-04-22-hp-server-nvram-corruption-emergency.md
|
||||
- clients/valleywide/session-logs/2026-05-12-session.md
|
||||
- clients/valleywide/docs/yealink-phones.md
|
||||
- clients/valleywide/docs/yealink-t54w-recovery-procedure.md
|
||||
- clients/valleywide/app-modernization/CONTEXT.md
|
||||
- clients/valleywide/app-modernization/session-logs/2026-04-27-session.md
|
||||
- clients/valleywide/app-modernization/research/schema-analysis.md
|
||||
- clients/valleywide/app-modernization/source-analysis/D-drive-2026-05-16/SUMMARY.md
|
||||
- clients/valleywide/app-modernization/source-analysis/drive2-2026-05-16/SUMMARY.md
|
||||
- clients/valleywide/app-modernization/source-analysis/drive3-2026-05-16/SUMMARY.md
|
||||
backlinks: []
|
||||
---
|
||||
|
||||
# Valley Wide Plastering
|
||||
|
||||
Plastering / stucco subcontractor based in Arizona. Active ACG client. Primary work has been incident response (RDWeb brute-force, power outage recovery) and an ongoing app modernization project for their custom VB6/Access construction ERP.
|
||||
|
||||
---
|
||||
|
||||
## Profile
|
||||
|
||||
- **Company type:** Construction subcontractor (plastering / stucco)
|
||||
- **Domain / site identifier:** VWP (`vwp.local` internal AD domain, `vwp.us` registered external domain, `valleywideplastering.com` M365 domain)
|
||||
- **Contract type:** Prepaid hour block
|
||||
- **Hours remaining:** 10.0 hrs as of 2026-05-12 (after billing 1.5 hrs for HP server emergency). Always live-check Syncro before billing.
|
||||
- **Billing rate:** $150/hr remote labor (`product 1190473 — Labor - Remote Business`)
|
||||
- **Emergency surcharge pattern:** Bill as two line items — 1.0 hr normal + 0.5 hr surcharge. Use product 1190473 for both (NOT product 26184, which bakes in a 1.5x dollar rate that would double-charge prepaid block customers). Results in 1.5 hr block deduction = 150% charge.
|
||||
- **Key contact:** Shelly Dooley / Valley Wide P (Syncro customer display name)
|
||||
- **Syncro customer ID:** `31694734`
|
||||
- **Syncro ticket (2026-05-12 emergency):** #32269 (ID: `110159277`) — HP server powered off, ADSRVR unreachable. Invoiced; invoice #67594 (ID: `1650271395`). Ticket status: Invoiced.
|
||||
- **M365 tenant ID:** `5c53ae9f-7071-4248-b834-8685b646450f`
|
||||
- **M365 domain:** `valleywideplastering.com`
|
||||
|
||||
---
|
||||
|
||||
## Infrastructure
|
||||
|
||||
### Servers & Services
|
||||
|
||||
| Host | IP | Role | OS | Notes |
|
||||
|---|---|---|---|---|
|
||||
| HP ProLiant DL360 Gen10 (SN: MXQ80400X4) | (LAN — no static IP documented) | Hypervisor / VM host for ADSRVR | — | iLO at 172.16.9.125 (SSH port 22, legacy ssh-rsa key). Power outage 2026-04-22 caused NVRAM corruption + factory iLO reset. Was found powered-off 2026-05-12; powered on remotely via iLO. |
|
||||
| HP iLO | 172.16.9.125 | Out-of-band management for HP ProLiant | — | SSH port 22. **Requires legacy RSA algorithms** — modern OpenSSH rejects it. Use paramiko with `disabled_algorithms={'pubkeys': ['rsa-sha2-256', 'rsa-sha2-512']}`. Credentials in vault: `clients/valleywide/` |
|
||||
| VWP_ADSRVR | 192.168.0.25 | Domain Controller for `vwp.local` | Windows Server 2019 Standard (build 17763) | VM on HP ProLiant DL360 Gen10. SSH enabled, key auth working for `vwp\guru` (ed25519, added 2026-04-13). Default shell is cmd.exe — use `powershell -NoProfile -Command` wrappers. |
|
||||
| VWP-QBS | 172.16.9.169 | QuickBooks server + RDS/RemoteApp host | Windows Server 2022 Standard | **Physical Dell server** (NOT a VM). Has DRAC. Runs IIS (RD Web Access, RD Gateway). Reach from ADSRVR via `Invoke-Command -ComputerName VWP-QBS -Credential` with `vwp\sysadmin` PSCredential — no direct SSH; Kerberos does not forward over SSH double-hop. WinRM on 5985. |
|
||||
| Dell DRAC (VWP-QBS) | [undocumented] | Out-of-band management for VWP-QBS Dell | — | DRAC functional as of 2026-04-22; used to force manual boot after power outage. IP not yet documented. |
|
||||
| DC1 | 172.16.9.2 | Domain Controller | — | Confirmed up 2026-05-12. Separate from ADSRVR. |
|
||||
| XenServer (older Dell) | 192.168.0.104 | VM hypervisor — hosts BACKUP-SRV, Server 2012 R2, Server 2003 | XenServer | Older Dell hardware. Was offline after 2026-04-22 power outage; status resolved. Credentials: `root` / see vault. |
|
||||
| UDM (UniFi Dream Machine) | 172.16.9.1 | Perimeter firewall, OpenVPN server, DHCP, DNS, site router | UniFi OS | DNS override: `vwp-qbs.vwp.us` → 172.16.9.169 (static record in UDM dnsmasq). VPN pushes DNS=192.168.4.1 (UDM). WireGuard site-to-site peers present (wgsts1001, wgsts1003, wgsts1005 — likely UniFi SiteMagic). |
|
||||
|
||||
**[WARNING] No UPS on HP ProLiant DL360.** The 2026-04-22 power outage caused NVRAM corruption. A UPS assessment is an outstanding priority item — hardware failure from power event is a proven risk.
|
||||
|
||||
### Email & Identity
|
||||
|
||||
- **M365 tenant:** `valleywideplastering.com` | Tenant ID: `5c53ae9f-7071-4248-b834-8685b646450f`
|
||||
- **On-prem AD domain:** `vwp.local` (internal). External registered domain: `vwp.us` (used for internal FQDNs like `vwp-qbs.vwp.us`).
|
||||
- **MFA status:** [unverified] — No M365 CA or MFA configuration documented. Not investigated.
|
||||
- **MX / mail flow:** [unverified] — M365 tenant confirmed but mail flow not audited.
|
||||
|
||||
### Network
|
||||
|
||||
- **ISP / WAN:** Public WAN IP `98.168.18.21` (observed via Yealink YMCS last-seen registrar)
|
||||
- **Firewall / Router:** UniFi Dream Machine at 172.16.9.1
|
||||
- **VPN:** OpenVPN on UDM. Client pool: `192.168.4.0/24`. Pushes routes for `172.16.9.0/24`, `192.168.0.0/24`, `192.168.3.0/24`. DNS pushed as `192.168.4.1` (UDM).
|
||||
- **Subnets:**
|
||||
- `172.16.9.0/24` — primary internal network (servers, Dell VWP-QBS, UDM, iLO)
|
||||
- `192.168.0.0/24` — secondary internal (AD server, Yealink phones) [WARNING: conflicts with IMC's LAN — be careful when switching VPN contexts between clients]
|
||||
- `192.168.4.0/24` — OpenVPN client pool
|
||||
- **Static DNS (UDM):** `vwp-qbs.vwp.us` → `172.16.9.169` (fixed typo from `qwp-qbs.vwp.us` on 2026-04-16)
|
||||
|
||||
### RDS / RemoteApp
|
||||
|
||||
- **Session host:** VWP-QBS (Windows Server 2022)
|
||||
- **Mode:** VPN-only (direct connect, no RD Gateway). Gateway was removed from the deployment 2026-04-16 after the RDWeb public exposure was closed. RDP manifests write `gatewayusagemethod:i:0`.
|
||||
- **RDS Licensing:** Per User mode. License server pointed at `vwp-qbs.vwp.us` (the same box — RDS-Licensing role was installed and activated on 2026-04-16 but had no real CALs).
|
||||
- **[WARNING] RDS CALs not purchased.** VWP-QBS license server has only the `Built-in TS Per Device CAL` placeholder. Users will start seeing "no licenses available" errors once grace period expires. Action: purchase Windows Server 2022 RDS Per User CALs, sized to active user count (check distinct interactive logons last 30 days via `licmgr.msc`).
|
||||
- **Application:** QuickBooks RemoteApp. VPN clients resolve `vwp-qbs.vwp.us` via UDM dnsmasq override and connect directly.
|
||||
|
||||
### Voice / IP Phones
|
||||
|
||||
- **Fleet:** 16x Yealink SIP-T54W color IP phones (OUIs `805e0c` and `44dbd2`)
|
||||
- **YMCS portal:** https://us.ymcs.yealink.com/manager/sip-product/sipManage — account: Valleywide Plastering (VWP)
|
||||
- **YMCS admin password:** vault — `clients/valleywide/` (Yealink password documented 2026-04-22)
|
||||
- **Status as of 2026-04-22:** 5 phones previously provisioned (Offline in YMCS), 11 pending first boot
|
||||
- **Named phones:** `214-ValleyWidePlastering` (extension 214), `Reception` (front desk, 192.168.0.17)
|
||||
- **Phone subnet:** `192.168.0.0/24` — phones on DHCP, IPs observed at .17, .54, .130, .140, .222
|
||||
- **[WARNING] Known-bad firmware:** `96.86.0.20` is a documented T54W brick-maker. Confirm YMCS firmware policy is NOT pushing this version before any mass provisioning.
|
||||
- **Recovery procedure:** TFTP recovery documented in `clients/valleywide/docs/yealink-t54w-recovery-procedure.md`. Use Tftpd64 with laptop at `192.168.81.100`, phone at `192.168.81.10`. Multiple recovery file sets may be needed (NEW RM → OLD RM → SPEAKER variant).
|
||||
|
||||
---
|
||||
|
||||
## Access
|
||||
|
||||
- **SSH to VWP_ADSRVR:** `ssh vwp\guru@192.168.0.25` (ed25519 key auth — key added 2026-04-13)
|
||||
- **Double-hop to VWP-QBS:** Via WinRM — `Invoke-Command -ComputerName VWP-QBS -Credential $cred` using `vwp\sysadmin` PSCredential from ADSRVR. SSH won't forward Kerberos for domain double-hop.
|
||||
- **HP iLO power management:** Paramiko required (not system OpenSSH). SSH to `172.16.9.125:22`. Use `disabled_algorithms={'pubkeys': ['rsa-sha2-256', 'rsa-sha2-512']}`. Command: `start system1` to power on.
|
||||
- **VWP-QBS DRAC:** IP undocumented — needs to be recorded. DRAC functional.
|
||||
- **VPN:** Connect to VWP OpenVPN (UDM) first; this provides access to both the 172.16.9.0/24 and 192.168.0.0/24 subnets.
|
||||
- **Vault paths:** `clients/valleywide/` (confirmed entries: `adsrvr`, `dc1`, `udm`, `xenserver`, `quickbooks-server-idrac`). Access via `bash "$VAULT" get-field clients/valleywide/<entry> <field>`.
|
||||
|
||||
---
|
||||
|
||||
## App Modernization Project
|
||||
|
||||
VWP's core business application is a custom-built construction ERP. The original developer (known as "Darv") is deceased. The app is hitting the 2GB Jet/Access database file size limit. ACG was engaged to assess modernization feasibility.
|
||||
|
||||
### Application Stack (Confirmed)
|
||||
|
||||
| Layer | Technology | Evidence |
|
||||
|---|---|---|
|
||||
| Frontend / logic | Visual Basic 6.0 | `frmPayroll.frm` source file, `.frx` resource files, `VB5!` header in exe |
|
||||
| Compilation | **P-Code** (not Native Code) | Entry point `PUSH+CALL` to ThunRTMain by ordinal — not native binary |
|
||||
| Database | MS Access Jet 3.x (.mdb) | `VWP.mdb` version byte 0x00, Access 97 format |
|
||||
| Reporting | Crystal Reports 8.5 | 791 `.rpt` files (per 2026-04-27 archive); Crystl32.OCX import; SCR85Dev installer found |
|
||||
| Installer | InstallShield Denali 2021 | `Denali2021v1` folder on server |
|
||||
| OCX controls | TABCTL32, mscomct2, comdlg32, Flp32a30, odg7, todg7 | PE import table |
|
||||
|
||||
**P-Code is the best possible outcome for decompilation.** VB Decompiler Pro (~$200) can recover 70-80% of source including form layouts, procedure names, string literals, and all SQL queries. Decompilation was approved as the next step.
|
||||
|
||||
### Database: VWP.mdb
|
||||
|
||||
- **Current size:** 938 MB (last written 2026-04-24). Growth: 671 MB (2020) → 761 MB (2022) → 938 MB (2026). **Approaching the 2 GB Jet hard limit.**
|
||||
- **Format:** Jet 3.x / Access 97. Modern ACE/DAO drivers refuse to open it — binary scan was used for schema extraction.
|
||||
- **Scale:** ~130 production tables spanning a full construction ERP.
|
||||
|
||||
#### Domain Coverage
|
||||
|
||||
| Domain | Key Tables |
|
||||
|---|---|
|
||||
| Projects & Jobs | tblPROJECT, tblLOTINFO, tblPLANS, tblCHANGE, tblSZONE |
|
||||
| Work Orders & Estimating | tblORDERS, tblTAKE, tblMEASURE, tblPlanBill |
|
||||
| Inventory & Purchasing | tblINVPRICE, tblINVTRY, tblSUPPLIER, tblPOrder, tblYardOrder |
|
||||
| Crew & Payroll | tblCREW, tblHRDAILY, tblPAYHEADER, tblPAYROLL, tblCREWRATE |
|
||||
| **Certified Payroll** | **tblCERTIFIED** — government / prevailing wage work. **HARD requirement.** |
|
||||
| Accounts Receivable | tblARMASTER, tblARINVOICE, tblARTRANS |
|
||||
| Accounts Payable | tblAPMASTER, tblAPTRANS, tblJOBCOST, tblCHECKREC |
|
||||
| **Positive Pay (3 banks)** | **tblPosPayVWP, tblPosPayCRD, tblPosPaySWI** — fraud-prevention bank integration. **HARD dependency.** |
|
||||
| Scaffold | tblScaffold, tblSC_Crew |
|
||||
| Repairs | tblREPAIR, tblRepList |
|
||||
| System / Config | tblSECURITY, tblSYSInfo, tblGLAcct |
|
||||
|
||||
**Modernization complexity: HIGH.** 791 Crystal Reports files, certified payroll (legal compliance — cannot be dropped), positive pay integration with 3 banks, and full AR/AP/Payroll.
|
||||
|
||||
### Source Code Status
|
||||
|
||||
The production exe (`Orders_10A.exe`, 13.4 MB) has four shortcuts pointing to it. The original source was on Darv's personal development machine — only one form file (`frmPayroll.frm`, 32 KB) was found on the server at `C:\Users\sysadmin\Desktop\Darv\Source\VWP\`. The remainder of `C:\Users\sysadmin\Desktop\Darv\` (13,231 files, 15.6 GB) includes Darv's installer projects, Crystal Reports, and personal files. VB6 source (`.vbp`, `.frm`) was scanned across multiple server drives (D: and two additional drives as of 2026-05-16). Substantial VB6 source exists across the drives (thousands of `.frm` and `.vbp` files); Mike was searching to confirm which are for the VWP application specifically.
|
||||
|
||||
### Project Status (as of 2026-04-27)
|
||||
|
||||
| Task | Status |
|
||||
|---|---|
|
||||
| Stack identification | Complete — VB6 P-Code + Jet 3.x confirmed |
|
||||
| Schema mapping (table names) | Complete (~130 tables via binary scan) |
|
||||
| Full schema with field types | Pending — needs Access 97/2000 environment or Jet 3.x → Jet 4.x conversion |
|
||||
| VB6 source search across server drives | In progress — Mike searching |
|
||||
| VB Decompiler Pro purchase and run | Pending ($200 investment) |
|
||||
| Crystal Reports audit (791 .rpt files) | Pending |
|
||||
| VWP staff workflow interviews | Pending |
|
||||
| Feasibility / modernization report | Pending |
|
||||
|
||||
---
|
||||
|
||||
## Patterns & Known Issues
|
||||
|
||||
### iLO Access (Non-Standard)
|
||||
|
||||
The HP ProLiant iLO at 172.16.9.125 uses legacy SSH host key algorithms (`ssh-rsa`/`ssh-dss`) that are rejected by modern OpenSSH on Windows by default. **Do not use system OpenSSH to connect.** Use Python paramiko with:
|
||||
|
||||
```python
|
||||
transport.disabled_algorithms = {'pubkeys': ['rsa-sha2-256', 'rsa-sha2-512']}
|
||||
```
|
||||
|
||||
Power-on command: `start system1`.
|
||||
|
||||
### RDS Double-Hop Pattern
|
||||
|
||||
SSH to ADSRVR (192.168.0.25) works fine with ed25519 key. But you cannot forward Kerberos over SSH to reach VWP-QBS — the WinRM double-hop must be done inside the SSH session using explicit PSCredential:
|
||||
|
||||
```powershell
|
||||
$cred = Get-Credential # vwp\sysadmin
|
||||
Invoke-Command -ComputerName VWP-QBS -Credential $cred -ScriptBlock { ... }
|
||||
```
|
||||
|
||||
### 192.168.0.0/24 Subnet Conflict
|
||||
|
||||
VWP's AD/phone subnet (`192.168.0.0/24`) is the same RFC1918 range as IMC (another ACG client). When switching between client VPN contexts, verify which 192.168.0.x addresses are being targeted. This is a silent risk — wrong subnet = wrong client.
|
||||
|
||||
### Syncro Billing for Prepaid Block Emergency
|
||||
|
||||
Do not use product 26184 (Labor - Emergency) for prepaid block customers. That product has the 1.5x rate baked in, which would result in double-charging when combined with the surcharge line item pattern. Always use product 1190473 for both normal and surcharge line items.
|
||||
|
||||
### AD Account: `scanner`
|
||||
|
||||
The `scanner` AD account is used by some device or process (original purpose unknown). Its password was last set 2024-10-17. During the 2026-04-13 brute-force incident, it was being locked out every ~20 minutes by attacker attempts through the public-facing RDWeb. **Password rotation is an outstanding hygiene item.**
|
||||
|
||||
### LastLogonDate Anomaly
|
||||
|
||||
VWP-QBS AD object showed `LastLogonDate: 9/28/2049` — flagged as a time-skew artifact during 2026-04-13 incident. Likely cosmetic.
|
||||
|
||||
---
|
||||
|
||||
## Active Work (as of 2026-05-12)
|
||||
|
||||
| Item | Status | Priority |
|
||||
|---|---|---|
|
||||
| App modernization: VB Decompiler Pro run against Orders_10A.exe | Pending — decompiler not yet purchased | High |
|
||||
| App modernization: Full schema extraction with field types | Pending — needs Access 97/2000 environment | High |
|
||||
| App modernization: VB6 source search across server drives | In progress | High |
|
||||
| RDS CAL purchase (Windows Server 2022 Per User, sized to user count) | Outstanding — grace period may expire | High |
|
||||
| HP iLO reconfiguration (post factory-reset 2026-04-22) | [unverified — may have been configured during 2026-04-22 onsite; confirm credentials in vault] | Medium |
|
||||
| UPS assessment for HP ProLiant | Outstanding since 2026-04-22 | Medium |
|
||||
| Yealink phone fleet provisioning (11 pending phones) | Outstanding — 11 of 16 phones never connected to YMCS | Medium |
|
||||
| `scanner` AD account password rotation | Outstanding since 2026-04-13 | Low |
|
||||
| UDM UPnP audit | Outstanding since 2026-04-13 | Low |
|
||||
| DRAC IP documentation for VWP-QBS | Not yet recorded | Low |
|
||||
|
||||
---
|
||||
|
||||
## Security Posture
|
||||
|
||||
### 2026-04-13: RDWeb Brute-Force Incident
|
||||
|
||||
RDWeb (`https://VWP-QBS/RDWeb/Pages/login.aspx`) was publicly exposed via UDM port-forward on port 443. A distributed brute-force botnet (residential proxy infrastructure, IPs from China, Belarus, UAE, and others) was hammering `POST /RDWeb/Pages/en-US/login.aspx` at ~6 req/min, hitting usernames `scanner`, `Guest`, `Receptionist`. This triggered AD lockouts every ~20 minutes (lockout threshold 5, 16-min window) which initially appeared to be a stale internal credential problem.
|
||||
|
||||
**Resolution:** UDM port-forward removed (same day), IIS reset to drain in-flight sessions, lockout policy restored. 30-day audit of Event 4624 confirmed **zero successful external logons — no compromise**.
|
||||
|
||||
**Current state:** RDWeb accessible from VPN and internal LAN only (port 443 on VWP-QBS, 172.16.9.0/24). Not reachable from public internet.
|
||||
|
||||
**Outstanding recommendation:** If RDWeb must be re-exposed publicly, require: IPBan (https://github.com/DigitalRuby/IPBan), firewall restriction to known source IPs, and 2FA/Conditional Access.
|
||||
|
||||
### 2026-04-22: Power Outage / NVRAM Corruption
|
||||
|
||||
Power outage caused HP ProLiant NVRAM corruption (BIOS/iLO factory reset). VWP-QBS Dell server had a boot retry loop (resolved via DRAC). XenServer (older Dell) was offline. All recovered onsite. **Root cause: no UPS on HP server.**
|
||||
|
||||
---
|
||||
|
||||
## History Highlights
|
||||
|
||||
| Date | Event |
|
||||
|---|---|
|
||||
| 2026-04-13 | RDWeb brute-force incident discovered and contained. SSH key deployed to ADSRVR. 30-day audit — no compromise. |
|
||||
| 2026-04-13 | Domain lockout policy temporarily disabled during diagnosis (threshold=0), restored to 5/16min/16min. 15-minute window of reduced lockout protection. |
|
||||
| 2026-04-16 | RDS reconfigured to VPN-only (gateway removed). UDM DNS typo fixed (`qwp-qbs` → `vwp-qbs`). RDS licensing mode set Per User, pointed at local license server. |
|
||||
| 2026-04-22 | Emergency onsite: power outage, HP ProLiant NVRAM corruption + iLO factory reset, VWP-QBS boot loop (DRAC), XenServer offline. All resolved ~12:00 MST. |
|
||||
| 2026-04-22 | Yealink SIP-T54W fleet (16 devices) added to YMCS device management. 5 previously-provisioned, 11 pending. |
|
||||
| 2026-04-27 | App modernization project initiated. VB6 P-Code + Jet 3.x stack confirmed. ~130 table schema extracted via binary scan. Crystal Reports 8.5 (791 .rpt files) documented. |
|
||||
| 2026-05-12 | HP ProLiant found powered-off (ADSRVR unreachable). Powered on remotely via iLO paramiko. Syncro ticket #32269, invoice #67594, 1.5 hr block deduction (10.0 hrs remaining). |
|
||||
|
||||
---
|
||||
|
||||
## Compilation Notes
|
||||
|
||||
**Date range covered:** 2026-04-13 through 2026-05-12.
|
||||
|
||||
**Items flagged [unverified]:**
|
||||
- M365 MFA and mail flow configuration — never investigated
|
||||
- HP iLO credentials post factory-reset — should be confirmed via vault; iLO was accessible 2026-05-12 so credentials were re-established at some point
|
||||
- XenServer resolution detail after 2026-04-22 outage — session log notes it offline/critical, subsequent sessions confirm it was up by 2026-05-12
|
||||
- DRAC IP for VWP-QBS — functional but undocumented
|
||||
- Yealink provisioning status — 11 phones still pending as of 2026-04-22; no follow-up session
|
||||
- RDS CAL grace period expiry timing — unknown; may have already expired
|
||||
@@ -20,6 +20,8 @@ Run `/wiki-lint` to check for stale entries and broken backlinks.
|
||||
|---|---|---|
|
||||
| [Cascades of Tucson](clients/cascades-tucson.md) | Prepaid block $175/hr, ~37.5 hrs remaining; senior living; active domain migration + HIPAA compliance project; single DC on aging R610 hardware | 2026-05-24 |
|
||||
| [Dataforth Corporation](clients/dataforth.md) | Prepaid block ~$2,099/mo; signal conditioning manufacturer; 64 DOS test stations; 2025 crypto attack recovery; 2026-03-27 phishing incident + MFA rollout; active test datasheet pipeline project; Neptune Exchange colocated at D2 | 2026-05-24 |
|
||||
| [Instrumental Music Center](clients/instrumental-music-center.md) | Prepaid block $175/hr, 12.5 hrs remaining; music retail/repair; AIMsi POS on SQL Server 2019; phantom DC causing slow logons; GuruRMM enrolled (IMC1) | 2026-05-24 |
|
||||
| [Valley Wide Plastering](clients/valleywide.md) | Prepaid block, 10 hrs remaining; plastering/stucco contractor; HP DL360 Gen10 + XenServer; VB6 app modernization project; RDWeb brute-force incident; 11 Yealink phones pending | 2026-05-24 |
|
||||
|
||||
## Projects
|
||||
|
||||
@@ -52,6 +54,8 @@ Run `/wiki-lint` to check for stale entries and broken backlinks.
|
||||
| Cascades of Tucson | CS-SERVER (192.168.2.254), pfSense (192.168.0.1), cascadesDS (192.168.0.120) | GuruRMM (RECEPTIONIST-PC + CS-SERVER enrolled) |
|
||||
| ACG Internal | gururmm-build (172.16.3.30), Jupiter (172.16.3.20), Pluto (172.16.3.36), Uranus (172.16.3.21) | GuruRMM server + ClaudeTools API on gururmm-build; Windows MSI builds on Pluto; Gitea/NPM/Seafile on Jupiter. Saturn DECOMMISSIONED. |
|
||||
| Dataforth Corporation | AD1 (192.168.0.27), AD2 (192.168.0.6), D2TESTNAS (192.168.0.9), SAGE-SQL (192.168.0.153), UDM (192.168.0.254); Neptune Exchange physically at Dataforth D2 (172.16.3.11 / 67.206.163.124) | Dataforth DOS — Test Datasheet Pipeline; GuruRMM (DF-GAGETRAK enrolled) |
|
||||
| Instrumental Music Center | IMC1 (192.168.0.2), phantom DC ServerIMC (192.168.0.63 — DNS-only, do not use) | GuruRMM (IMC1 enrolled) |
|
||||
| Valley Wide Plastering | VWP_ADSRVR (192.168.0.25), VWP-QBS (172.16.9.169), HP DL360 iLO (172.16.9.125), UDM (172.16.9.1) | — |
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user