diff --git a/wiki/clients/instrumental-music-center.md b/wiki/clients/instrumental-music-center.md new file mode 100644 index 0000000..a350c06 --- /dev/null +++ b/wiki/clients/instrumental-music-center.md @@ -0,0 +1,325 @@ +--- +type: client +name: instrumental-music-center +display_name: Instrumental Music Center +last_compiled: 2026-05-24 +compiled_by: DESKTOP-0O8A1RL/claude-main +sources: + - clients/instrumental-music-center/README.md + - clients/instrumental-music-center/PROJECT_STATE.md + - clients/instrumental-music-center/docs/overview.md + - clients/instrumental-music-center/docs/billing-log.md + - clients/instrumental-music-center/docs/2026-04-13-ticket-notes.md + - clients/instrumental-music-center/docs/network/topology.md + - clients/instrumental-music-center/docs/network/vlans.md + - clients/instrumental-music-center/docs/network/firewall.md + - clients/instrumental-music-center/docs/network/dhcp.md + - clients/instrumental-music-center/docs/network/dns.md + - clients/instrumental-music-center/docs/cloud/m365.md + - clients/instrumental-music-center/docs/cloud/azure.md + - clients/instrumental-music-center/docs/rmm/rmm.md + - clients/instrumental-music-center/docs/security/antivirus.md + - clients/instrumental-music-center/docs/security/backup.md + - clients/instrumental-music-center/docs/issues/log.md + - clients/instrumental-music-center/docs/servers/server_template.md + - clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md + - clients/instrumental-music-center/session-logs/2026-04-28-howard-manda-laptop-provision.md + - clients/instrumental-music-center/session-logs/2026-05-04-station2-printer-and-manda-vpn.md + - clients/instrumental-music-center/session-logs/2026-05-05-howard-aim-connection-broken-investigation.md + - clients/instrumental-music-center/session-logs/2026-05-06-howard-imc1-aim-instance-correction.md + - clients/instrumental-music-center/decisions/2026-05-07-mike-memory-allocation-approval.md +backlinks: + - projects/gururmm +--- + +# Instrumental Music Center + +Music retail and instrument repair shop running AIMsi point-of-sale software on-prem. Single-site as far as documented. Located at 7063 E Speedway Blvd, Tucson AZ 85710. ACG provides managed break-fix / prepaid-block support; primary focus is on the AIMsi SQL server (IMC1) and workstation fleet. + +--- + +## Profile + +- **Contract type:** Prepaid hour block +- **Billing rate:** $175/hr all labor +- **Hours remaining:** 12.5 hrs as of 2026-04-28 (after debiting 1.5 hrs for Syncro #32218). Always live-check before billing. +- **Syncro customer ID:** 7088508 +- **Key contacts:** + - **Leslie Stirm** — primary contact; leslie@imc-az.com; Syncro contact_id 731730 + - **Manda** — General Manager (new, replaced Michael Santander as of ~2026-04-28). Full name unconfirmed in AD. [unverified] + - **Michael Santander** — former GM; domain account already deactivated. +- **Primary domain:** imc.local (on-prem AD) +- **Location:** Speedway (7063 E Speedway Blvd, Tucson AZ 85710) — additional locations TBD; only Speedway is documented. +- **Critical software:** AIMsi by Tri-Tech (https://www.tritechretail.com/topic/aim) — retail POS and inventory management. + +--- + +## Infrastructure + +### Servers & Services + +| Host | IP | Role | OS | Notes | +|---|---|---|---|---| +| IMC1 | 192.168.0.2 | DC (imc.local), DNS, File Server, AIMsi SQL host, RDS host | Windows Server 2016 Standard (build 14393.7426) | Dell R720, 4 physical cores, 32 GB RAM. GuruRMM agent: `fa99e913-1027-4e33-a928-7695e31068e7` | +| ServerIMC | 192.168.0.63 | Phantom / broken DC | Windows Server 2016 Essentials [unverified] | **[WARNING] Registered as DC in AD DNS (A + SRV records for `_ldap._tcp.dc._msdcs.imc.local` and `_kerberos._tcp.imc.local`). Responds to ICMP but TCP/389 (LDAP) and TCP/88 (Kerberos) refuse connections. DC locator round-robins — clients that pick ServerIMC time out. Root cause of intermittent slow logons, GPO failures, and 2026-04-22 remote domain-join failure. Needs `ntdsutil` metadata cleanup (if demoted ghost) or AD service repair.** | +| IMC2 | — | Unknown (stale) | Windows Server 2016 Essentials | Last logon 2023 — likely decommissioned. Clean up AD computer object. | +| IMC-VM | — | Unknown (dead) | Windows Server 2016 Standard | Last logon 2021 — dead. Clean up AD computer object. | +| Station 1 | 192.168.0.50 | POS workstation | Windows [unverified] | Hostname `IMC-STATION1`. Primary workstation for AIM "connection broken" incidents. | + +#### IMC1 SQL Instances (CRITICAL — read carefully) + +**[WARNING] The production AIM database is on `IMC1\SQLEXPRESS`, NOT `IMC1\AIMSQL`. The instance name is actively misleading — someone installed SQL Server 2019 Standard under the default `SQLEXPRESS` instance name and never renamed it. This burned a full day of triage. Always verify SQL roles by active connections (`sys.dm_exec_sessions`) — never by instance name.** + +| Instance | Port | Edition (actual) | Role | Production DB | Notes | +|---|---|---|---|---|---| +| `IMC1\SQLEXPRESS` | TCP 61151 | **SQL Server 2019 Standard** (misleading name!) | **PRODUCTION** | `IMCAIM` (created 2023-08-21) | Service account `IMC\AIM`. ~9 store workstations + 22 server-local AIM sessions. **Do not stop, do not uninstall.** ERRORLOG at `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Log\`. No `max server memory` cap (default unlimited). | +| `IMC1\AIMSQL` | TCP 63116 (dynamic) | SQL Server 2019 Express GDR 15.0.2165.1 | **Orphan** (consolidation candidate) | None active | Service account `IMC\IMC1$`. Zero established TCP connections. Holds only 2023-era conversion-test DBs (`AIM`, `IMC`, `TestConv61223`). No active backup chain landing here. Shutdown + uninstall approved by Mike pending `.mdf` backup confirmation. | +| `IMC1\MICROSOFT##WID` | — | Windows Internal Database | WSUS / AD RMS | — | WSUS confirmed NOT in use at IMC. AD RMS status unverified. If AD RMS also unused, instance can be stopped to free ~300 MB. **Canary for memory pressure** — Event 17890 paging events fire here first when the host is memory-squeezed. | + +**Workstations connected to production `IMC1\SQLEXPRESS` (verified 2026-05-06):** + +| Hostname | IP | +|---|---| +| IMC-MINI | 192.168.0.72 | +| IMC-SVCSTR | 192.168.0.55 | +| IMC-LESSONS | 192.168.0.62 | +| IMC-STATION2 | 192.168.0.66 | +| IMC-L1-STATION9 | 192.168.0.41 | +| DESKTOP-44L80C0 | 192.168.0.46 | +| DESKTOP-MR3ALTK | 192.168.0.59 | +| REPAIRADMIN | 192.168.0.48 | +| C2B | 192.168.0.4 | +| IMC-STATION1 | 192.168.0.50 | + +All sessions authenticate as `AIMUser1` via `.Net SqlClient Data Provider`. + +#### IMC1 Disk Layout + +| Drive | Purpose | Notes | +|---|---|---| +| C: | OS, IIS, system DBs | 419 GB volume; ~278 GB used after 2026-04-12 cleanup (~66%); was 77% full before. Monitor. | +| E: | SQL backups + installers + Server 2016 media | `E:\W2016\sources\install.wim` is RTM 14393.0. SQL backups at `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\` | +| F: | Windows Image Backups | — | +| S: | Dedicated SSD (Samsung 850 PRO 256 GB) — AIMsi SQL user DBs | User DBs at `S:\SQL\Data\`. AIM client share `\\IMC1\AIM` → `S:\AIM`. System DBs remain on C:. | + +### Email & Identity + +- **Mail:** IMC uses a **mixed Google / Microsoft identity model** — different users are on different platforms. Manda is on the M365 side. [full tenant details unverified] +- **M365 tenant details:** Not fully documented. Manda's Outlook was configured against an existing M365 mailbox. +- **On-prem AD domain:** `imc.local` +- **MFA status:** [unverified] +- **DNS:** IMC1 (192.168.0.2) is the authoritative DNS server for imc.local. ServerIMC (192.168.0.63) has ghost A + SRV records as a DC — these are the direct cause of client authentication failures and need cleanup. + +### Network + +- **LAN subnet:** 192.168.0.0/24 +- **VPN:** OpenVPN (.ovpn profile). **[WARNING] 192.168.0.0/24 subnet overlap hazard:** if technician's home/office LAN is also 192.168.0.0/24 (Howard's home is), OpenVPN routes win for reaching IMC1 but Windows multi-homed DNS races between the two interfaces. DNS negative caching causes domain join / locator failures. **If remote LAN overlaps IMC's subnet, go onsite for domain joins.** Also: disconnect Tailscale before connecting to IMC OpenVPN — Tailscale's `pfsense-2` subnet router advertises 192.168.0.0/24 with lower metric than the VPN, making IMC1 unreachable. +- **Firewall:** [unverified — not documented] +- **ISP:** [unverified] +- **SMB:** SMB1 still enabled on IMC1 — disable as security hygiene when opportunity permits. +- **SMB signing:** `RequireSecuritySignature = True` on server — adds auth overhead. + +--- + +## GuruRMM Enrollment + +| Field | Value | +|---|---| +| GuruRMM client | Instrumental Music Center | +| GuruRMM client ID | `213b62a8-30f4-41dd-9bb3-549341104416` | +| GuruRMM client code | `IMC` | +| Site | IMCMain | +| Site ID | `2c5b65ad-2d5e-47b3-b12b-632e35e08ff6` | +| Site code | `INNER-BRIDGE-8354` | +| Site enrollment key | vault: `clients/imc/gururmm-site-main.sops.yaml` | +| First enrolled agent | IMC1 (`fa99e913-1027-4e33-a928-7695e31068e7`) | + +IMC was enrolled in GuruRMM on 2026-05-05 (Howard, prompted by AIM connection-broken investigation). IMC1 agent was installed by Mike via ScreenConnect. Only IMC1 is enrolled as of last session — workstations not yet enrolled. + +**Note:** When SSH from Howard-Home is blocked by the 192.168.0.0/24 route collision, GuruRMM remote commands are the fallback for running diagnostics on IMC1. + +--- + +## Access + +- **SSH:** `ssh IMC\guru@192.168.0.2` — ed25519 key auth; PowerShell is the default shell. Authorized keys: `C:\ProgramData\ssh\administrators_authorized_keys` (inheritance off, Administrators + SYSTEM full control). +- **VPN:** OpenVPN (.ovpn profile). Disconnect Tailscale first. If home/office LAN is 192.168.0.0/24, remote domain operations will fail — go onsite instead. +- **Domain admin:** `IMC\guru` — also SQL sysadmin on both SQLEXPRESS and AIMSQL (added via single-user recovery 2026-04-12). +- **GuruRMM:** IMC1 agent `fa99e913-1027-4e33-a928-7695e31068e7` — use for remote commands when SSH is blocked. +- **Vault paths:** + - IMC1 credentials (domain admin, SSH): `clients/imc/imc1.sops.yaml` + - GuruRMM site enrollment key: `clients/imc/gururmm-site-main.sops.yaml` + +**[WARNING] `sa` account on AIMSQL:** exists and enabled; password unknown. One candidate was tried and failed on 2026-04-12 — no lockout triggered (no lockout policy). If needed for AIMSQL consolidation, use single-user recovery mode (same process used 2026-04-12). + +--- + +## AIMsi / Tri-Tech Critical Notes + +**Per-machine workstation number (`USER#`) is mandatory.** AIMsi requires a user environment variable `USER#` (older Tri-Tech convention, still in use at IMC) set on each machine. This is the per-machine workstation identifier for POS polling and licensing. + +- **NEVER wipe or reimage a machine without recording its `USER#` first.** +- **When deploying a new machine, assign its `USER#` per Leslie** — she tracks the allocation. +- Tri-Tech docs: https://www.tritechretail.com/topic/aim + +**Known `USER#` assignments:** + +| Machine | Hostname | USER# | Notes | +|---|---|---|---| +| Manda (GM) laptop | DESKTOP-KRHQ5TS | 4 | Assigned per Leslie, 2026-04-28 | +| Other workstations | Various | TBD | Not yet fully documented | + +--- + +## Backups + +- **Local SQL backups:** Nightly at 22:00 to `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\IMCAIM_*.bak` +- **Retention script:** `C:\Scripts\Clean-AimsiBackups.ps1` — GFS policy: 14 dailies + 1st-of-month; 3-newest safety override; logs to `C:\Scripts\Logs\aimsi-retention-YYYYMM.log` +- **Retention task:** `IMC AIMsi Backup Retention` — daily 23:30, SYSTEM, 1-hour limit +- **Off-site:** Cloudberry / MSP360 at `C:\ProgramData\Online Backup\`. Cloudberry chain confirmed intact before 2026-04-12 deletion run. + - SQLEXPRESS backup also confirmed landing at `C:\ProgramData\Online Backup\MSSQL\IMC1_SQLEXPRESS\` +- **Windows Image Backup:** on F: +- **AIMSQL orphan:** no backup chain. Locate and back up `AIM.mdf`, `IMC.mdf`, `TestConv61223.mdf` and their `.ldf` siblings before any consolidation — files were not found in expected path under `MSSQL15.AIMSQL\MSSQL\DATA` or `S:\*AIMSQL*` during 2026-05-06 search. + +--- + +## Patterns & Known Issues + +### [WARNING] Phantom DC `ServerIMC` — Active Authentication Degrader + +`ServerIMC` (192.168.0.63) is registered in DNS as a domain controller (A record + SRV records for `_ldap._tcp.dc._msdcs.imc.local` and `_kerberos._tcp.imc.local`) alongside IMC1. It responds to ICMP ping but TCP/389 and TCP/88 refuse connections. The DC locator round-robins between IMC1 and ServerIMC, timing out ~50% of the time. + +**Effect:** Intermittent slow logons, GPO failures, and broken remote domain joins for every domain client at IMC. Was the confirmed root cause of the 2026-04-22 failed remote join of `DESKTOP-KRHQ5TS`. + +**Action needed:** Open a ticket. Either: +1. Repair AD services if `ServerIMC` is a real machine with broken services, or +2. Run `ntdsutil` metadata cleanup if it is a ghost from a previously demoted DC. + +This was first flagged as "unclear" on 2026-04-13, promoted to confirmed issue 2026-04-28. No ticket has been opened as of 2026-05-06. + +### AIM "Connection Broken" — Memory Pressure on IMC1 + +**Symptom:** `Telerik.OpenAccess.RT.sql.SQLException: Connection has been closed / The connection is broken and recovery is not possible` — user-facing AIM crash. First seen 2026-05-05 on Station 1 (IMC-STATION1, 192.168.0.50), recurred 2026-05-06 ~12:14 PM. + +**Root cause:** IMC1 is hosting DC services + 6 concurrent RDP users + AIMsi Webservice/Runtime + three SQL instances + QuickBooks Enterprise on 32 GB. Under memory pressure, Windows trims SQL working sets (visible as WID Event 17890 paging events — the canary). The trim reaps idle Telerik OpenAccess TCP pool slots. Telerik has no transient-fault retry, so the next query against a dead pool handle throws the raw stack trace. + +**SQLEXPRESS has no `max server memory` cap** (default 2,147,483,647 MB). Working set observed at 6.86 GB. + +**Approved fix (Mike, 2026-05-07):** Cap `max server memory` on each instance: +- `SQLEXPRESS`: 12,288 MB (12 GB) +- `MSSQL$MICROSOFT##WID`: 512 MB +- `MSSQL$AIMSQL`: 256 MB (or consolidate it) + +**Status as of 2026-05-06:** Howard is awaiting go-ahead for implementation. Mike approved on 2026-05-07. **Confirm whether Howard has applied the caps — this is the immediate recurrence prevention.** [unverified post-2026-05-07] + +### [WARNING] SQL Instance Name Trap + +**`IMC1\SQLEXPRESS` is SQL Server 2019 Standard Edition** — someone installed Standard under the default `SQLEXPRESS` instance name and never renamed it. `SERVERPROPERTY('Edition')` is the only way to confirm this. The instance name actively misleads. + +**Never assume an instance is idle, orphan, or Express based on name.** Always verify by: +1. `SERVERPROPERTY('Edition')` for edition +2. `sys.dm_exec_sessions` for active user sessions +3. `Get-NetTCPConnection -OwningProcess` for established TCP connections + +This trap caused a wrong-instance restart task to be deployed (2026-05-05) that had zero effect on the user-facing problem and was unregistered the next day (2026-05-06). See `.claude/memory/feedback_sql_instance_role_by_connection.md`. + +### Component Store Corruption on IMC1 (RDS Removal Blocked) + +`COMPONENTS` registry hive is ~168 MB (normal 30-50 MB), causing `0x80073701 ERROR_SXS_ASSEMBLY_MISSING` on any role removal or CU apply-on-boot. ETW manifest for provider GUID `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}` is malformed — causes `CBS_E_INSTALLERS_FAILED` → full rollback even when CU staging succeeds. + +**Effect:** Blocks RDS role removal, which was the original reason for the 2026-04-12 engagement. Also means CU KB5075999 cannot be applied cleanly. + +**Server is otherwise healthy** — AIMsi production is running. This is a structural impediment to the Server 2019 migration. Three paths considered (see History Highlights). + +### Remote Domain Join Over OpenVPN — Don't Do It + +If the technician's local LAN subnet overlaps IMC's 192.168.0.0/24, remote domain joins over OpenVPN will fail reliably: +- OpenVPN pushed routes win for TCP, but Windows multi-homed DNS races between LAN DNS and VPN DNS (both respond to `imc.local` queries; LAN returns NXDOMAIN faster; Windows caches the negative answer). +- Even with NRPT rules, hosts file entries, `-Server ` on Add-Computer, and `nltest /dsgetdc /force` — the combination of subnet overlap + phantom DC (ServerIMC) beat all client-side workarounds. + +**Rule:** For IMC domain operations where local subnet overlap exists, go onsite. + +### Mixed Email Identity (Google + M365) + +IMC users are split between Google Workspace and Microsoft 365 — different users on different platforms. When configuring a new user, confirm with Leslie which platform their mailbox lives on before setting up Outlook vs. Gmail. + +### Stale AD Objects + +| Object | Last Logon | Status | Action | +|---|---|---|---| +| IMC2 (computer) | 2023 | Likely decommissioned | Clean up AD object | +| IMC-VM (computer) | 2021 | Dead | Clean up AD object | +| ServerIMC (DC) | Active (ICMP) | Phantom/broken DC | ntdsutil metadata cleanup or repair | + +### GPO Noise + +- **DistributedCOM 10016** fires every 5 minutes — RuntimeBroker permission noise. Cosmetic. +- **Group Policy event 103** fires every 5 minutes — "removal of the assignment of application Syncro from policy Management SW failed". Stale GPO object. Cleanup separately. + +### Server 2016 EOL + +Extended support ends **2027-01-12**. Migration window is finite. The memory pressure / AIM reliability incident is additional evidence to push the migration timeline. Mike wants to scope cost/timeline at next ACG strategy call. + +--- + +## Active Work + +As of 2026-05-07 (last decision recorded): + +1. **[IMMEDIATE] Apply `max server memory` caps on IMC1 SQL instances** — Mike approved 2026-05-07. Howard to implement: SQLEXPRESS 12 GB, WID 512 MB, AIMSQL 256 MB. Reversible (1-second config change, no service restart). Until applied, AIM connection-broken errors will continue recurring. [unverified — confirm applied] + +2. **[HIGH] Open ticket for ServerIMC phantom DC investigation** — SRV/A records in DNS claim it's a DC; LDAP/Kerberos refuse connections. Degrades authentication for all domain users. No ticket opened as of 2026-05-06. + +3. **[MEDIUM] AIMSQL orphan consolidation** — Mike approved (2026-05-07). Pending: + - Locate `AIM.mdf`, `IMC.mdf`, `TestConv61223.mdf` and `.ldf` siblings (not in expected path) + - Back up 2023-era DBs before shutdown + - Verify no applications reference `IMC1\AIMSQL` (TCP 63116) + - Stop and uninstall `MSSQL$AIMSQL` + +4. **[MEDIUM] WID instance decision** — Verify AD RMS usage. WSUS confirmed unused. If AD RMS also unused, stop WID to free ~300 MB headroom. Mike awaiting Howard's verification before authorizing stop. + +5. **[LOWER] Server 2019 migration scoping** — Three paths (component store repair + in-place; in-place without repair; clean build). Clean build is Mike's recommendation. Scope cost/timeline at next ACG strategy call before 2027-01-12 EOL. + +6. **[LOWER] Documentation cleanup:** + - Update workstation table in `docs/overview.md` with `DESKTOP-KRHQ5TS` / Manda / AIM USER#=4 + - Confirm Manda's full name in AD + - Disable SMB1 on IMC1 (`Set-SmbServerConfiguration -EnableSMB1Protocol $false`) + - Drop `TestConv61223` DB on AIMSQL (leftover 2023 migration test) — safe per enumeration, but back up `.mdf` first + - Clean up stale AD computer objects `IMC2`, `IMC-VM` + +--- + +## History Highlights + +| Date | By | Event | +|---|---|---| +| ~2026-Q1 | Mike/Howard | Early engagement: 3 new workstations provisioned at Speedway (hostnames, AIM USER#s TBD in billing log) | +| 2026-04-11/12/13 | Mike | IMC1 maintenance: RDS removal blocked (component store corruption 0x80073701), SSH installed, 716 GB freed on E: (backup cleanup), GFS retention automated, AIMsi DBs moved C:→S: SSD | +| 2026-04-22 | Howard | Attempted remote domain-join of `DESKTOP-KRHQ5TS` over VPN — abandoned after subnet overlap + phantom DC defeated all workarounds | +| 2026-04-28 | Howard | Onsite: `DESKTOP-KRHQ5TS` joined to imc.local, Manda (new GM) AD account created, Outlook/M365 configured, Office activated, AIMsi USER#=4 set. Ticket #32218, 1.5 hrs, prepay 14.0→12.5 hrs. ServerIMC confirmed as active authentication degrader. | +| 2026-05-04 | Howard | Onsite (0.5 hrs): Station 2 receipt printer reconnected (re-added from \\imc1); VPN installed on Manda's machine. Ticket #32247. | +| 2026-05-05 | Howard | AIM "connection broken" investigation. GuruRMM IMC client/site provisioned, IMC1 enrolled. Diagnosed memory pressure; scheduled AIMSQL restart for 02:30 (wrong instance — superseded next day). | +| 2026-05-06 | Howard | Station 1 recurrence 12:14 PM. Full instance enumeration revealed SQLEXPRESS = production Standard (not AIMSQL). Wrong-instance restart task unregistered. Corrected diagnosis in session logs and PROJECT_STATE. Feedback memory created. | +| 2026-05-07 | Mike | Decision: approved memory caps (SQLEXPRESS 12 GB, WID 512 MB, AIMSQL 256 MB), AIMSQL consolidation pending backup, Server 2016 migration timeline acknowledged, WSUS confirmed unused. | + +--- + +## Compilation Notes + +Source material: 5 session logs (2026-04-12 through 2026-05-06) + 1 decision file (2026-05-07) + README + PROJECT_STATE + 10 docs files (most docs/* are blank templates with no client-specific data filled in — network/firewall/vlans/VLAN/DHCP/DNS/RMM/AV/backup/issues docs are all empty templates). + +Many structured docs (`docs/network/`, `docs/security/`, `docs/cloud/`) are empty templates. The authoritative information sources are `README.md`, `PROJECT_STATE.md`, and the session logs. + +**Unverified items flagged:** +- Whether Howard applied `max server memory` caps after Mike's 2026-05-07 approval +- ServerIMC ticket status — ticket was recommended but not confirmed opened +- Manda's full name in AD +- M365 tenant details (tenant domain, license type, MFA policy) +- WID instance AD RMS usage +- AIMSQL `.mdf` file locations +- Full workstation fleet AIM USER# assignments +- ISP, firewall hardware, VLAN/network topology + +## Backlinks + +- [[projects/gururmm]] — IMC1 enrolled as agent `fa99e913-1027-4e33-a928-7695e31068e7`; site IMCMain diff --git a/wiki/clients/valleywide.md b/wiki/clients/valleywide.md new file mode 100644 index 0000000..4595ff2 --- /dev/null +++ b/wiki/clients/valleywide.md @@ -0,0 +1,271 @@ +--- +type: client +name: valleywide +display_name: Valley Wide Plastering +last_compiled: 2026-05-24 +compiled_by: DESKTOP-0O8A1RL/claude-main +sources: + - clients/valleywide/README.md + - clients/valleywide/PROJECT_STATE.md + - clients/valleywide/session-logs/2026-04-13-rdweb-brute-force-incident.md + - clients/valleywide/session-logs/2026-04-22-hp-server-nvram-corruption-emergency.md + - clients/valleywide/session-logs/2026-05-12-session.md + - clients/valleywide/docs/yealink-phones.md + - clients/valleywide/docs/yealink-t54w-recovery-procedure.md + - clients/valleywide/app-modernization/CONTEXT.md + - clients/valleywide/app-modernization/session-logs/2026-04-27-session.md + - clients/valleywide/app-modernization/research/schema-analysis.md + - clients/valleywide/app-modernization/source-analysis/D-drive-2026-05-16/SUMMARY.md + - clients/valleywide/app-modernization/source-analysis/drive2-2026-05-16/SUMMARY.md + - clients/valleywide/app-modernization/source-analysis/drive3-2026-05-16/SUMMARY.md +backlinks: [] +--- + +# Valley Wide Plastering + +Plastering / stucco subcontractor based in Arizona. Active ACG client. Primary work has been incident response (RDWeb brute-force, power outage recovery) and an ongoing app modernization project for their custom VB6/Access construction ERP. + +--- + +## Profile + +- **Company type:** Construction subcontractor (plastering / stucco) +- **Domain / site identifier:** VWP (`vwp.local` internal AD domain, `vwp.us` registered external domain, `valleywideplastering.com` M365 domain) +- **Contract type:** Prepaid hour block +- **Hours remaining:** 10.0 hrs as of 2026-05-12 (after billing 1.5 hrs for HP server emergency). Always live-check Syncro before billing. +- **Billing rate:** $150/hr remote labor (`product 1190473 — Labor - Remote Business`) +- **Emergency surcharge pattern:** Bill as two line items — 1.0 hr normal + 0.5 hr surcharge. Use product 1190473 for both (NOT product 26184, which bakes in a 1.5x dollar rate that would double-charge prepaid block customers). Results in 1.5 hr block deduction = 150% charge. +- **Key contact:** Shelly Dooley / Valley Wide P (Syncro customer display name) +- **Syncro customer ID:** `31694734` +- **Syncro ticket (2026-05-12 emergency):** #32269 (ID: `110159277`) — HP server powered off, ADSRVR unreachable. Invoiced; invoice #67594 (ID: `1650271395`). Ticket status: Invoiced. +- **M365 tenant ID:** `5c53ae9f-7071-4248-b834-8685b646450f` +- **M365 domain:** `valleywideplastering.com` + +--- + +## Infrastructure + +### Servers & Services + +| Host | IP | Role | OS | Notes | +|---|---|---|---|---| +| HP ProLiant DL360 Gen10 (SN: MXQ80400X4) | (LAN — no static IP documented) | Hypervisor / VM host for ADSRVR | — | iLO at 172.16.9.125 (SSH port 22, legacy ssh-rsa key). Power outage 2026-04-22 caused NVRAM corruption + factory iLO reset. Was found powered-off 2026-05-12; powered on remotely via iLO. | +| HP iLO | 172.16.9.125 | Out-of-band management for HP ProLiant | — | SSH port 22. **Requires legacy RSA algorithms** — modern OpenSSH rejects it. Use paramiko with `disabled_algorithms={'pubkeys': ['rsa-sha2-256', 'rsa-sha2-512']}`. Credentials in vault: `clients/valleywide/` | +| VWP_ADSRVR | 192.168.0.25 | Domain Controller for `vwp.local` | Windows Server 2019 Standard (build 17763) | VM on HP ProLiant DL360 Gen10. SSH enabled, key auth working for `vwp\guru` (ed25519, added 2026-04-13). Default shell is cmd.exe — use `powershell -NoProfile -Command` wrappers. | +| VWP-QBS | 172.16.9.169 | QuickBooks server + RDS/RemoteApp host | Windows Server 2022 Standard | **Physical Dell server** (NOT a VM). Has DRAC. Runs IIS (RD Web Access, RD Gateway). Reach from ADSRVR via `Invoke-Command -ComputerName VWP-QBS -Credential` with `vwp\sysadmin` PSCredential — no direct SSH; Kerberos does not forward over SSH double-hop. WinRM on 5985. | +| Dell DRAC (VWP-QBS) | [undocumented] | Out-of-band management for VWP-QBS Dell | — | DRAC functional as of 2026-04-22; used to force manual boot after power outage. IP not yet documented. | +| DC1 | 172.16.9.2 | Domain Controller | — | Confirmed up 2026-05-12. Separate from ADSRVR. | +| XenServer (older Dell) | 192.168.0.104 | VM hypervisor — hosts BACKUP-SRV, Server 2012 R2, Server 2003 | XenServer | Older Dell hardware. Was offline after 2026-04-22 power outage; status resolved. Credentials: `root` / see vault. | +| UDM (UniFi Dream Machine) | 172.16.9.1 | Perimeter firewall, OpenVPN server, DHCP, DNS, site router | UniFi OS | DNS override: `vwp-qbs.vwp.us` → 172.16.9.169 (static record in UDM dnsmasq). VPN pushes DNS=192.168.4.1 (UDM). WireGuard site-to-site peers present (wgsts1001, wgsts1003, wgsts1005 — likely UniFi SiteMagic). | + +**[WARNING] No UPS on HP ProLiant DL360.** The 2026-04-22 power outage caused NVRAM corruption. A UPS assessment is an outstanding priority item — hardware failure from power event is a proven risk. + +### Email & Identity + +- **M365 tenant:** `valleywideplastering.com` | Tenant ID: `5c53ae9f-7071-4248-b834-8685b646450f` +- **On-prem AD domain:** `vwp.local` (internal). External registered domain: `vwp.us` (used for internal FQDNs like `vwp-qbs.vwp.us`). +- **MFA status:** [unverified] — No M365 CA or MFA configuration documented. Not investigated. +- **MX / mail flow:** [unverified] — M365 tenant confirmed but mail flow not audited. + +### Network + +- **ISP / WAN:** Public WAN IP `98.168.18.21` (observed via Yealink YMCS last-seen registrar) +- **Firewall / Router:** UniFi Dream Machine at 172.16.9.1 +- **VPN:** OpenVPN on UDM. Client pool: `192.168.4.0/24`. Pushes routes for `172.16.9.0/24`, `192.168.0.0/24`, `192.168.3.0/24`. DNS pushed as `192.168.4.1` (UDM). +- **Subnets:** + - `172.16.9.0/24` — primary internal network (servers, Dell VWP-QBS, UDM, iLO) + - `192.168.0.0/24` — secondary internal (AD server, Yealink phones) [WARNING: conflicts with IMC's LAN — be careful when switching VPN contexts between clients] + - `192.168.4.0/24` — OpenVPN client pool +- **Static DNS (UDM):** `vwp-qbs.vwp.us` → `172.16.9.169` (fixed typo from `qwp-qbs.vwp.us` on 2026-04-16) + +### RDS / RemoteApp + +- **Session host:** VWP-QBS (Windows Server 2022) +- **Mode:** VPN-only (direct connect, no RD Gateway). Gateway was removed from the deployment 2026-04-16 after the RDWeb public exposure was closed. RDP manifests write `gatewayusagemethod:i:0`. +- **RDS Licensing:** Per User mode. License server pointed at `vwp-qbs.vwp.us` (the same box — RDS-Licensing role was installed and activated on 2026-04-16 but had no real CALs). +- **[WARNING] RDS CALs not purchased.** VWP-QBS license server has only the `Built-in TS Per Device CAL` placeholder. Users will start seeing "no licenses available" errors once grace period expires. Action: purchase Windows Server 2022 RDS Per User CALs, sized to active user count (check distinct interactive logons last 30 days via `licmgr.msc`). +- **Application:** QuickBooks RemoteApp. VPN clients resolve `vwp-qbs.vwp.us` via UDM dnsmasq override and connect directly. + +### Voice / IP Phones + +- **Fleet:** 16x Yealink SIP-T54W color IP phones (OUIs `805e0c` and `44dbd2`) +- **YMCS portal:** https://us.ymcs.yealink.com/manager/sip-product/sipManage — account: Valleywide Plastering (VWP) +- **YMCS admin password:** vault — `clients/valleywide/` (Yealink password documented 2026-04-22) +- **Status as of 2026-04-22:** 5 phones previously provisioned (Offline in YMCS), 11 pending first boot +- **Named phones:** `214-ValleyWidePlastering` (extension 214), `Reception` (front desk, 192.168.0.17) +- **Phone subnet:** `192.168.0.0/24` — phones on DHCP, IPs observed at .17, .54, .130, .140, .222 +- **[WARNING] Known-bad firmware:** `96.86.0.20` is a documented T54W brick-maker. Confirm YMCS firmware policy is NOT pushing this version before any mass provisioning. +- **Recovery procedure:** TFTP recovery documented in `clients/valleywide/docs/yealink-t54w-recovery-procedure.md`. Use Tftpd64 with laptop at `192.168.81.100`, phone at `192.168.81.10`. Multiple recovery file sets may be needed (NEW RM → OLD RM → SPEAKER variant). + +--- + +## Access + +- **SSH to VWP_ADSRVR:** `ssh vwp\guru@192.168.0.25` (ed25519 key auth — key added 2026-04-13) +- **Double-hop to VWP-QBS:** Via WinRM — `Invoke-Command -ComputerName VWP-QBS -Credential $cred` using `vwp\sysadmin` PSCredential from ADSRVR. SSH won't forward Kerberos for domain double-hop. +- **HP iLO power management:** Paramiko required (not system OpenSSH). SSH to `172.16.9.125:22`. Use `disabled_algorithms={'pubkeys': ['rsa-sha2-256', 'rsa-sha2-512']}`. Command: `start system1` to power on. +- **VWP-QBS DRAC:** IP undocumented — needs to be recorded. DRAC functional. +- **VPN:** Connect to VWP OpenVPN (UDM) first; this provides access to both the 172.16.9.0/24 and 192.168.0.0/24 subnets. +- **Vault paths:** `clients/valleywide/` (confirmed entries: `adsrvr`, `dc1`, `udm`, `xenserver`, `quickbooks-server-idrac`). Access via `bash "$VAULT" get-field clients/valleywide/ `. + +--- + +## App Modernization Project + +VWP's core business application is a custom-built construction ERP. The original developer (known as "Darv") is deceased. The app is hitting the 2GB Jet/Access database file size limit. ACG was engaged to assess modernization feasibility. + +### Application Stack (Confirmed) + +| Layer | Technology | Evidence | +|---|---|---| +| Frontend / logic | Visual Basic 6.0 | `frmPayroll.frm` source file, `.frx` resource files, `VB5!` header in exe | +| Compilation | **P-Code** (not Native Code) | Entry point `PUSH+CALL` to ThunRTMain by ordinal — not native binary | +| Database | MS Access Jet 3.x (.mdb) | `VWP.mdb` version byte 0x00, Access 97 format | +| Reporting | Crystal Reports 8.5 | 791 `.rpt` files (per 2026-04-27 archive); Crystl32.OCX import; SCR85Dev installer found | +| Installer | InstallShield Denali 2021 | `Denali2021v1` folder on server | +| OCX controls | TABCTL32, mscomct2, comdlg32, Flp32a30, odg7, todg7 | PE import table | + +**P-Code is the best possible outcome for decompilation.** VB Decompiler Pro (~$200) can recover 70-80% of source including form layouts, procedure names, string literals, and all SQL queries. Decompilation was approved as the next step. + +### Database: VWP.mdb + +- **Current size:** 938 MB (last written 2026-04-24). Growth: 671 MB (2020) → 761 MB (2022) → 938 MB (2026). **Approaching the 2 GB Jet hard limit.** +- **Format:** Jet 3.x / Access 97. Modern ACE/DAO drivers refuse to open it — binary scan was used for schema extraction. +- **Scale:** ~130 production tables spanning a full construction ERP. + +#### Domain Coverage + +| Domain | Key Tables | +|---|---| +| Projects & Jobs | tblPROJECT, tblLOTINFO, tblPLANS, tblCHANGE, tblSZONE | +| Work Orders & Estimating | tblORDERS, tblTAKE, tblMEASURE, tblPlanBill | +| Inventory & Purchasing | tblINVPRICE, tblINVTRY, tblSUPPLIER, tblPOrder, tblYardOrder | +| Crew & Payroll | tblCREW, tblHRDAILY, tblPAYHEADER, tblPAYROLL, tblCREWRATE | +| **Certified Payroll** | **tblCERTIFIED** — government / prevailing wage work. **HARD requirement.** | +| Accounts Receivable | tblARMASTER, tblARINVOICE, tblARTRANS | +| Accounts Payable | tblAPMASTER, tblAPTRANS, tblJOBCOST, tblCHECKREC | +| **Positive Pay (3 banks)** | **tblPosPayVWP, tblPosPayCRD, tblPosPaySWI** — fraud-prevention bank integration. **HARD dependency.** | +| Scaffold | tblScaffold, tblSC_Crew | +| Repairs | tblREPAIR, tblRepList | +| System / Config | tblSECURITY, tblSYSInfo, tblGLAcct | + +**Modernization complexity: HIGH.** 791 Crystal Reports files, certified payroll (legal compliance — cannot be dropped), positive pay integration with 3 banks, and full AR/AP/Payroll. + +### Source Code Status + +The production exe (`Orders_10A.exe`, 13.4 MB) has four shortcuts pointing to it. The original source was on Darv's personal development machine — only one form file (`frmPayroll.frm`, 32 KB) was found on the server at `C:\Users\sysadmin\Desktop\Darv\Source\VWP\`. The remainder of `C:\Users\sysadmin\Desktop\Darv\` (13,231 files, 15.6 GB) includes Darv's installer projects, Crystal Reports, and personal files. VB6 source (`.vbp`, `.frm`) was scanned across multiple server drives (D: and two additional drives as of 2026-05-16). Substantial VB6 source exists across the drives (thousands of `.frm` and `.vbp` files); Mike was searching to confirm which are for the VWP application specifically. + +### Project Status (as of 2026-04-27) + +| Task | Status | +|---|---| +| Stack identification | Complete — VB6 P-Code + Jet 3.x confirmed | +| Schema mapping (table names) | Complete (~130 tables via binary scan) | +| Full schema with field types | Pending — needs Access 97/2000 environment or Jet 3.x → Jet 4.x conversion | +| VB6 source search across server drives | In progress — Mike searching | +| VB Decompiler Pro purchase and run | Pending ($200 investment) | +| Crystal Reports audit (791 .rpt files) | Pending | +| VWP staff workflow interviews | Pending | +| Feasibility / modernization report | Pending | + +--- + +## Patterns & Known Issues + +### iLO Access (Non-Standard) + +The HP ProLiant iLO at 172.16.9.125 uses legacy SSH host key algorithms (`ssh-rsa`/`ssh-dss`) that are rejected by modern OpenSSH on Windows by default. **Do not use system OpenSSH to connect.** Use Python paramiko with: + +```python +transport.disabled_algorithms = {'pubkeys': ['rsa-sha2-256', 'rsa-sha2-512']} +``` + +Power-on command: `start system1`. + +### RDS Double-Hop Pattern + +SSH to ADSRVR (192.168.0.25) works fine with ed25519 key. But you cannot forward Kerberos over SSH to reach VWP-QBS — the WinRM double-hop must be done inside the SSH session using explicit PSCredential: + +```powershell +$cred = Get-Credential # vwp\sysadmin +Invoke-Command -ComputerName VWP-QBS -Credential $cred -ScriptBlock { ... } +``` + +### 192.168.0.0/24 Subnet Conflict + +VWP's AD/phone subnet (`192.168.0.0/24`) is the same RFC1918 range as IMC (another ACG client). When switching between client VPN contexts, verify which 192.168.0.x addresses are being targeted. This is a silent risk — wrong subnet = wrong client. + +### Syncro Billing for Prepaid Block Emergency + +Do not use product 26184 (Labor - Emergency) for prepaid block customers. That product has the 1.5x rate baked in, which would result in double-charging when combined with the surcharge line item pattern. Always use product 1190473 for both normal and surcharge line items. + +### AD Account: `scanner` + +The `scanner` AD account is used by some device or process (original purpose unknown). Its password was last set 2024-10-17. During the 2026-04-13 brute-force incident, it was being locked out every ~20 minutes by attacker attempts through the public-facing RDWeb. **Password rotation is an outstanding hygiene item.** + +### LastLogonDate Anomaly + +VWP-QBS AD object showed `LastLogonDate: 9/28/2049` — flagged as a time-skew artifact during 2026-04-13 incident. Likely cosmetic. + +--- + +## Active Work (as of 2026-05-12) + +| Item | Status | Priority | +|---|---|---| +| App modernization: VB Decompiler Pro run against Orders_10A.exe | Pending — decompiler not yet purchased | High | +| App modernization: Full schema extraction with field types | Pending — needs Access 97/2000 environment | High | +| App modernization: VB6 source search across server drives | In progress | High | +| RDS CAL purchase (Windows Server 2022 Per User, sized to user count) | Outstanding — grace period may expire | High | +| HP iLO reconfiguration (post factory-reset 2026-04-22) | [unverified — may have been configured during 2026-04-22 onsite; confirm credentials in vault] | Medium | +| UPS assessment for HP ProLiant | Outstanding since 2026-04-22 | Medium | +| Yealink phone fleet provisioning (11 pending phones) | Outstanding — 11 of 16 phones never connected to YMCS | Medium | +| `scanner` AD account password rotation | Outstanding since 2026-04-13 | Low | +| UDM UPnP audit | Outstanding since 2026-04-13 | Low | +| DRAC IP documentation for VWP-QBS | Not yet recorded | Low | + +--- + +## Security Posture + +### 2026-04-13: RDWeb Brute-Force Incident + +RDWeb (`https://VWP-QBS/RDWeb/Pages/login.aspx`) was publicly exposed via UDM port-forward on port 443. A distributed brute-force botnet (residential proxy infrastructure, IPs from China, Belarus, UAE, and others) was hammering `POST /RDWeb/Pages/en-US/login.aspx` at ~6 req/min, hitting usernames `scanner`, `Guest`, `Receptionist`. This triggered AD lockouts every ~20 minutes (lockout threshold 5, 16-min window) which initially appeared to be a stale internal credential problem. + +**Resolution:** UDM port-forward removed (same day), IIS reset to drain in-flight sessions, lockout policy restored. 30-day audit of Event 4624 confirmed **zero successful external logons — no compromise**. + +**Current state:** RDWeb accessible from VPN and internal LAN only (port 443 on VWP-QBS, 172.16.9.0/24). Not reachable from public internet. + +**Outstanding recommendation:** If RDWeb must be re-exposed publicly, require: IPBan (https://github.com/DigitalRuby/IPBan), firewall restriction to known source IPs, and 2FA/Conditional Access. + +### 2026-04-22: Power Outage / NVRAM Corruption + +Power outage caused HP ProLiant NVRAM corruption (BIOS/iLO factory reset). VWP-QBS Dell server had a boot retry loop (resolved via DRAC). XenServer (older Dell) was offline. All recovered onsite. **Root cause: no UPS on HP server.** + +--- + +## History Highlights + +| Date | Event | +|---|---| +| 2026-04-13 | RDWeb brute-force incident discovered and contained. SSH key deployed to ADSRVR. 30-day audit — no compromise. | +| 2026-04-13 | Domain lockout policy temporarily disabled during diagnosis (threshold=0), restored to 5/16min/16min. 15-minute window of reduced lockout protection. | +| 2026-04-16 | RDS reconfigured to VPN-only (gateway removed). UDM DNS typo fixed (`qwp-qbs` → `vwp-qbs`). RDS licensing mode set Per User, pointed at local license server. | +| 2026-04-22 | Emergency onsite: power outage, HP ProLiant NVRAM corruption + iLO factory reset, VWP-QBS boot loop (DRAC), XenServer offline. All resolved ~12:00 MST. | +| 2026-04-22 | Yealink SIP-T54W fleet (16 devices) added to YMCS device management. 5 previously-provisioned, 11 pending. | +| 2026-04-27 | App modernization project initiated. VB6 P-Code + Jet 3.x stack confirmed. ~130 table schema extracted via binary scan. Crystal Reports 8.5 (791 .rpt files) documented. | +| 2026-05-12 | HP ProLiant found powered-off (ADSRVR unreachable). Powered on remotely via iLO paramiko. Syncro ticket #32269, invoice #67594, 1.5 hr block deduction (10.0 hrs remaining). | + +--- + +## Compilation Notes + +**Date range covered:** 2026-04-13 through 2026-05-12. + +**Items flagged [unverified]:** +- M365 MFA and mail flow configuration — never investigated +- HP iLO credentials post factory-reset — should be confirmed via vault; iLO was accessible 2026-05-12 so credentials were re-established at some point +- XenServer resolution detail after 2026-04-22 outage — session log notes it offline/critical, subsequent sessions confirm it was up by 2026-05-12 +- DRAC IP for VWP-QBS — functional but undocumented +- Yealink provisioning status — 11 phones still pending as of 2026-04-22; no follow-up session +- RDS CAL grace period expiry timing — unknown; may have already expired diff --git a/wiki/index.md b/wiki/index.md index c72cbaf..7ea576b 100644 --- a/wiki/index.md +++ b/wiki/index.md @@ -20,6 +20,8 @@ Run `/wiki-lint` to check for stale entries and broken backlinks. |---|---|---| | [Cascades of Tucson](clients/cascades-tucson.md) | Prepaid block $175/hr, ~37.5 hrs remaining; senior living; active domain migration + HIPAA compliance project; single DC on aging R610 hardware | 2026-05-24 | | [Dataforth Corporation](clients/dataforth.md) | Prepaid block ~$2,099/mo; signal conditioning manufacturer; 64 DOS test stations; 2025 crypto attack recovery; 2026-03-27 phishing incident + MFA rollout; active test datasheet pipeline project; Neptune Exchange colocated at D2 | 2026-05-24 | +| [Instrumental Music Center](clients/instrumental-music-center.md) | Prepaid block $175/hr, 12.5 hrs remaining; music retail/repair; AIMsi POS on SQL Server 2019; phantom DC causing slow logons; GuruRMM enrolled (IMC1) | 2026-05-24 | +| [Valley Wide Plastering](clients/valleywide.md) | Prepaid block, 10 hrs remaining; plastering/stucco contractor; HP DL360 Gen10 + XenServer; VB6 app modernization project; RDWeb brute-force incident; 11 Yealink phones pending | 2026-05-24 | ## Projects @@ -52,6 +54,8 @@ Run `/wiki-lint` to check for stale entries and broken backlinks. | Cascades of Tucson | CS-SERVER (192.168.2.254), pfSense (192.168.0.1), cascadesDS (192.168.0.120) | GuruRMM (RECEPTIONIST-PC + CS-SERVER enrolled) | | ACG Internal | gururmm-build (172.16.3.30), Jupiter (172.16.3.20), Pluto (172.16.3.36), Uranus (172.16.3.21) | GuruRMM server + ClaudeTools API on gururmm-build; Windows MSI builds on Pluto; Gitea/NPM/Seafile on Jupiter. Saturn DECOMMISSIONED. | | Dataforth Corporation | AD1 (192.168.0.27), AD2 (192.168.0.6), D2TESTNAS (192.168.0.9), SAGE-SQL (192.168.0.153), UDM (192.168.0.254); Neptune Exchange physically at Dataforth D2 (172.16.3.11 / 67.206.163.124) | Dataforth DOS — Test Datasheet Pipeline; GuruRMM (DF-GAGETRAK enrolled) | +| Instrumental Music Center | IMC1 (192.168.0.2), phantom DC ServerIMC (192.168.0.63 — DNS-only, do not use) | GuruRMM (IMC1 enrolled) | +| Valley Wide Plastering | VWP_ADSRVR (192.168.0.25), VWP-QBS (172.16.9.169), HP DL360 iLO (172.16.9.125), UDM (172.16.9.1) | — | ---