--- type: client name: instrumental-music-center display_name: Instrumental Music Center last_compiled: 2026-05-24 compiled_by: DESKTOP-0O8A1RL/claude-main sources: - clients/instrumental-music-center/README.md - clients/instrumental-music-center/PROJECT_STATE.md - clients/instrumental-music-center/docs/overview.md - clients/instrumental-music-center/docs/billing-log.md - clients/instrumental-music-center/docs/2026-04-13-ticket-notes.md - clients/instrumental-music-center/docs/network/topology.md - clients/instrumental-music-center/docs/network/vlans.md - clients/instrumental-music-center/docs/network/firewall.md - clients/instrumental-music-center/docs/network/dhcp.md - clients/instrumental-music-center/docs/network/dns.md - clients/instrumental-music-center/docs/cloud/m365.md - clients/instrumental-music-center/docs/cloud/azure.md - clients/instrumental-music-center/docs/rmm/rmm.md - clients/instrumental-music-center/docs/security/antivirus.md - clients/instrumental-music-center/docs/security/backup.md - clients/instrumental-music-center/docs/issues/log.md - clients/instrumental-music-center/docs/servers/server_template.md - clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md - clients/instrumental-music-center/session-logs/2026-04-28-howard-manda-laptop-provision.md - clients/instrumental-music-center/session-logs/2026-05-04-station2-printer-and-manda-vpn.md - clients/instrumental-music-center/session-logs/2026-05-05-howard-aim-connection-broken-investigation.md - clients/instrumental-music-center/session-logs/2026-05-06-howard-imc1-aim-instance-correction.md - clients/instrumental-music-center/decisions/2026-05-07-mike-memory-allocation-approval.md backlinks: - projects/gururmm --- # Instrumental Music Center Music retail and instrument repair shop running AIMsi point-of-sale software on-prem. Single-site as far as documented. Located at 7063 E Speedway Blvd, Tucson AZ 85710. ACG provides managed break-fix / prepaid-block support; primary focus is on the AIMsi SQL server (IMC1) and workstation fleet. --- ## Profile - **Contract type:** Prepaid hour block - **Billing rate:** $175/hr all labor - **Hours remaining:** 12.5 hrs as of 2026-04-28 (after debiting 1.5 hrs for Syncro #32218). Always live-check before billing. - **Syncro customer ID:** 7088508 - **Key contacts:** - **Leslie Stirm** — primary contact; leslie@imc-az.com; Syncro contact_id 731730 - **Manda** — General Manager (new, replaced Michael Santander as of ~2026-04-28). Full name unconfirmed in AD. [unverified] - **Michael Santander** — former GM; domain account already deactivated. - **Primary domain:** imc.local (on-prem AD) - **Location:** Speedway (7063 E Speedway Blvd, Tucson AZ 85710) — additional locations TBD; only Speedway is documented. - **Critical software:** AIMsi by Tri-Tech (https://www.tritechretail.com/topic/aim) — retail POS and inventory management. --- ## Infrastructure ### Servers & Services | Host | IP | Role | OS | Notes | |---|---|---|---|---| | IMC1 | 192.168.0.2 | DC (imc.local), DNS, File Server, AIMsi SQL host, RDS host | Windows Server 2016 Standard (build 14393.7426) | Dell R720, 4 physical cores, 32 GB RAM. GuruRMM agent: `fa99e913-1027-4e33-a928-7695e31068e7` | | ServerIMC | 192.168.0.63 | Phantom / broken DC | Windows Server 2016 Essentials [unverified] | **[WARNING] Registered as DC in AD DNS (A + SRV records for `_ldap._tcp.dc._msdcs.imc.local` and `_kerberos._tcp.imc.local`). Responds to ICMP but TCP/389 (LDAP) and TCP/88 (Kerberos) refuse connections. DC locator round-robins — clients that pick ServerIMC time out. Root cause of intermittent slow logons, GPO failures, and 2026-04-22 remote domain-join failure. Needs `ntdsutil` metadata cleanup (if demoted ghost) or AD service repair.** | | IMC2 | — | Unknown (stale) | Windows Server 2016 Essentials | Last logon 2023 — likely decommissioned. Clean up AD computer object. | | IMC-VM | — | Unknown (dead) | Windows Server 2016 Standard | Last logon 2021 — dead. Clean up AD computer object. | | Station 1 | 192.168.0.50 | POS workstation | Windows [unverified] | Hostname `IMC-STATION1`. Primary workstation for AIM "connection broken" incidents. | #### IMC1 SQL Instances (CRITICAL — read carefully) **[WARNING] The production AIM database is on `IMC1\SQLEXPRESS`, NOT `IMC1\AIMSQL`. The instance name is actively misleading — someone installed SQL Server 2019 Standard under the default `SQLEXPRESS` instance name and never renamed it. This burned a full day of triage. Always verify SQL roles by active connections (`sys.dm_exec_sessions`) — never by instance name.** | Instance | Port | Edition (actual) | Role | Production DB | Notes | |---|---|---|---|---|---| | `IMC1\SQLEXPRESS` | TCP 61151 | **SQL Server 2019 Standard** (misleading name!) | **PRODUCTION** | `IMCAIM` (created 2023-08-21) | Service account `IMC\AIM`. ~9 store workstations + 22 server-local AIM sessions. **Do not stop, do not uninstall.** ERRORLOG at `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Log\`. No `max server memory` cap (default unlimited). | | `IMC1\AIMSQL` | TCP 63116 (dynamic) | SQL Server 2019 Express GDR 15.0.2165.1 | **Orphan** (consolidation candidate) | None active | Service account `IMC\IMC1$`. Zero established TCP connections. Holds only 2023-era conversion-test DBs (`AIM`, `IMC`, `TestConv61223`). No active backup chain landing here. Shutdown + uninstall approved by Mike pending `.mdf` backup confirmation. | | `IMC1\MICROSOFT##WID` | — | Windows Internal Database | WSUS / AD RMS | — | WSUS confirmed NOT in use at IMC. AD RMS status unverified. If AD RMS also unused, instance can be stopped to free ~300 MB. **Canary for memory pressure** — Event 17890 paging events fire here first when the host is memory-squeezed. | **Workstations connected to production `IMC1\SQLEXPRESS` (verified 2026-05-06):** | Hostname | IP | |---|---| | IMC-MINI | 192.168.0.72 | | IMC-SVCSTR | 192.168.0.55 | | IMC-LESSONS | 192.168.0.62 | | IMC-STATION2 | 192.168.0.66 | | IMC-L1-STATION9 | 192.168.0.41 | | DESKTOP-44L80C0 | 192.168.0.46 | | DESKTOP-MR3ALTK | 192.168.0.59 | | REPAIRADMIN | 192.168.0.48 | | C2B | 192.168.0.4 | | IMC-STATION1 | 192.168.0.50 | All sessions authenticate as `AIMUser1` via `.Net SqlClient Data Provider`. #### IMC1 Disk Layout | Drive | Purpose | Notes | |---|---|---| | C: | OS, IIS, system DBs | 419 GB volume; ~278 GB used after 2026-04-12 cleanup (~66%); was 77% full before. Monitor. | | E: | SQL backups + installers + Server 2016 media | `E:\W2016\sources\install.wim` is RTM 14393.0. SQL backups at `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\` | | F: | Windows Image Backups | — | | S: | Dedicated SSD (Samsung 850 PRO 256 GB) — AIMsi SQL user DBs | User DBs at `S:\SQL\Data\`. AIM client share `\\IMC1\AIM` → `S:\AIM`. System DBs remain on C:. | ### Email & Identity - **Mail:** IMC uses a **mixed Google / Microsoft identity model** — different users are on different platforms. Manda is on the M365 side. [full tenant details unverified] - **M365 tenant details:** Not fully documented. Manda's Outlook was configured against an existing M365 mailbox. - **On-prem AD domain:** `imc.local` - **MFA status:** [unverified] - **DNS:** IMC1 (192.168.0.2) is the authoritative DNS server for imc.local. ServerIMC (192.168.0.63) has ghost A + SRV records as a DC — these are the direct cause of client authentication failures and need cleanup. ### Network - **LAN subnet:** 192.168.0.0/24 - **VPN:** OpenVPN (.ovpn profile). **[WARNING] 192.168.0.0/24 subnet overlap hazard:** if technician's home/office LAN is also 192.168.0.0/24 (Howard's home is), OpenVPN routes win for reaching IMC1 but Windows multi-homed DNS races between the two interfaces. DNS negative caching causes domain join / locator failures. **If remote LAN overlaps IMC's subnet, go onsite for domain joins.** Also: disconnect Tailscale before connecting to IMC OpenVPN — Tailscale's `pfsense-2` subnet router advertises 192.168.0.0/24 with lower metric than the VPN, making IMC1 unreachable. - **Firewall:** [unverified — not documented] - **ISP:** [unverified] - **SMB:** SMB1 still enabled on IMC1 — disable as security hygiene when opportunity permits. - **SMB signing:** `RequireSecuritySignature = True` on server — adds auth overhead. --- ## GuruRMM Enrollment | Field | Value | |---|---| | GuruRMM client | Instrumental Music Center | | GuruRMM client ID | `213b62a8-30f4-41dd-9bb3-549341104416` | | GuruRMM client code | `IMC` | | Site | IMCMain | | Site ID | `2c5b65ad-2d5e-47b3-b12b-632e35e08ff6` | | Site code | `INNER-BRIDGE-8354` | | Site enrollment key | vault: `clients/imc/gururmm-site-main.sops.yaml` | | First enrolled agent | IMC1 (`fa99e913-1027-4e33-a928-7695e31068e7`) | IMC was enrolled in GuruRMM on 2026-05-05 (Howard, prompted by AIM connection-broken investigation). IMC1 agent was installed by Mike via ScreenConnect. Only IMC1 is enrolled as of last session — workstations not yet enrolled. **Note:** When SSH from Howard-Home is blocked by the 192.168.0.0/24 route collision, GuruRMM remote commands are the fallback for running diagnostics on IMC1. --- ## Access - **SSH:** `ssh IMC\guru@192.168.0.2` — ed25519 key auth; PowerShell is the default shell. Authorized keys: `C:\ProgramData\ssh\administrators_authorized_keys` (inheritance off, Administrators + SYSTEM full control). - **VPN:** OpenVPN (.ovpn profile). Disconnect Tailscale first. If home/office LAN is 192.168.0.0/24, remote domain operations will fail — go onsite instead. - **Domain admin:** `IMC\guru` — also SQL sysadmin on both SQLEXPRESS and AIMSQL (added via single-user recovery 2026-04-12). - **GuruRMM:** IMC1 agent `fa99e913-1027-4e33-a928-7695e31068e7` — use for remote commands when SSH is blocked. - **Vault paths:** - IMC1 credentials (domain admin, SSH): `clients/imc/imc1.sops.yaml` - GuruRMM site enrollment key: `clients/imc/gururmm-site-main.sops.yaml` **[WARNING] `sa` account on AIMSQL:** exists and enabled; password unknown. One candidate was tried and failed on 2026-04-12 — no lockout triggered (no lockout policy). If needed for AIMSQL consolidation, use single-user recovery mode (same process used 2026-04-12). --- ## AIMsi / Tri-Tech Critical Notes **Per-machine workstation number (`USER#`) is mandatory.** AIMsi requires a user environment variable `USER#` (older Tri-Tech convention, still in use at IMC) set on each machine. This is the per-machine workstation identifier for POS polling and licensing. - **NEVER wipe or reimage a machine without recording its `USER#` first.** - **When deploying a new machine, assign its `USER#` per Leslie** — she tracks the allocation. - Tri-Tech docs: https://www.tritechretail.com/topic/aim **Known `USER#` assignments:** | Machine | Hostname | USER# | Notes | |---|---|---|---| | Manda (GM) laptop | DESKTOP-KRHQ5TS | 4 | Assigned per Leslie, 2026-04-28 | | Other workstations | Various | TBD | Not yet fully documented | --- ## Backups - **Local SQL backups:** Nightly at 22:00 to `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\IMCAIM_*.bak` - **Retention script:** `C:\Scripts\Clean-AimsiBackups.ps1` — GFS policy: 14 dailies + 1st-of-month; 3-newest safety override; logs to `C:\Scripts\Logs\aimsi-retention-YYYYMM.log` - **Retention task:** `IMC AIMsi Backup Retention` — daily 23:30, SYSTEM, 1-hour limit - **Off-site:** Cloudberry / MSP360 at `C:\ProgramData\Online Backup\`. Cloudberry chain confirmed intact before 2026-04-12 deletion run. - SQLEXPRESS backup also confirmed landing at `C:\ProgramData\Online Backup\MSSQL\IMC1_SQLEXPRESS\` - **Windows Image Backup:** on F: - **AIMSQL orphan:** no backup chain. Locate and back up `AIM.mdf`, `IMC.mdf`, `TestConv61223.mdf` and their `.ldf` siblings before any consolidation — files were not found in expected path under `MSSQL15.AIMSQL\MSSQL\DATA` or `S:\*AIMSQL*` during 2026-05-06 search. --- ## Patterns & Known Issues ### [WARNING] Phantom DC `ServerIMC` — Active Authentication Degrader `ServerIMC` (192.168.0.63) is registered in DNS as a domain controller (A record + SRV records for `_ldap._tcp.dc._msdcs.imc.local` and `_kerberos._tcp.imc.local`) alongside IMC1. It responds to ICMP ping but TCP/389 and TCP/88 refuse connections. The DC locator round-robins between IMC1 and ServerIMC, timing out ~50% of the time. **Effect:** Intermittent slow logons, GPO failures, and broken remote domain joins for every domain client at IMC. Was the confirmed root cause of the 2026-04-22 failed remote join of `DESKTOP-KRHQ5TS`. **Action needed:** Open a ticket. Either: 1. Repair AD services if `ServerIMC` is a real machine with broken services, or 2. Run `ntdsutil` metadata cleanup if it is a ghost from a previously demoted DC. This was first flagged as "unclear" on 2026-04-13, promoted to confirmed issue 2026-04-28. No ticket has been opened as of 2026-05-06. ### AIM "Connection Broken" — Memory Pressure on IMC1 **Symptom:** `Telerik.OpenAccess.RT.sql.SQLException: Connection has been closed / The connection is broken and recovery is not possible` — user-facing AIM crash. First seen 2026-05-05 on Station 1 (IMC-STATION1, 192.168.0.50), recurred 2026-05-06 ~12:14 PM. **Root cause:** IMC1 is hosting DC services + 6 concurrent RDP users + AIMsi Webservice/Runtime + three SQL instances + QuickBooks Enterprise on 32 GB. Under memory pressure, Windows trims SQL working sets (visible as WID Event 17890 paging events — the canary). The trim reaps idle Telerik OpenAccess TCP pool slots. Telerik has no transient-fault retry, so the next query against a dead pool handle throws the raw stack trace. **SQLEXPRESS has no `max server memory` cap** (default 2,147,483,647 MB). Working set observed at 6.86 GB. **Approved fix (Mike, 2026-05-07):** Cap `max server memory` on each instance: - `SQLEXPRESS`: 12,288 MB (12 GB) - `MSSQL$MICROSOFT##WID`: 512 MB - `MSSQL$AIMSQL`: 256 MB (or consolidate it) **Status as of 2026-05-06:** Howard is awaiting go-ahead for implementation. Mike approved on 2026-05-07. **Confirm whether Howard has applied the caps — this is the immediate recurrence prevention.** [unverified post-2026-05-07] ### [WARNING] SQL Instance Name Trap **`IMC1\SQLEXPRESS` is SQL Server 2019 Standard Edition** — someone installed Standard under the default `SQLEXPRESS` instance name and never renamed it. `SERVERPROPERTY('Edition')` is the only way to confirm this. The instance name actively misleads. **Never assume an instance is idle, orphan, or Express based on name.** Always verify by: 1. `SERVERPROPERTY('Edition')` for edition 2. `sys.dm_exec_sessions` for active user sessions 3. `Get-NetTCPConnection -OwningProcess` for established TCP connections This trap caused a wrong-instance restart task to be deployed (2026-05-05) that had zero effect on the user-facing problem and was unregistered the next day (2026-05-06). See `.claude/memory/feedback_sql_instance_role_by_connection.md`. ### Component Store Corruption on IMC1 (RDS Removal Blocked) `COMPONENTS` registry hive is ~168 MB (normal 30-50 MB), causing `0x80073701 ERROR_SXS_ASSEMBLY_MISSING` on any role removal or CU apply-on-boot. ETW manifest for provider GUID `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}` is malformed — causes `CBS_E_INSTALLERS_FAILED` → full rollback even when CU staging succeeds. **Effect:** Blocks RDS role removal, which was the original reason for the 2026-04-12 engagement. Also means CU KB5075999 cannot be applied cleanly. **Server is otherwise healthy** — AIMsi production is running. This is a structural impediment to the Server 2019 migration. Three paths considered (see History Highlights). ### Remote Domain Join Over OpenVPN — Don't Do It If the technician's local LAN subnet overlaps IMC's 192.168.0.0/24, remote domain joins over OpenVPN will fail reliably: - OpenVPN pushed routes win for TCP, but Windows multi-homed DNS races between LAN DNS and VPN DNS (both respond to `imc.local` queries; LAN returns NXDOMAIN faster; Windows caches the negative answer). - Even with NRPT rules, hosts file entries, `-Server ` on Add-Computer, and `nltest /dsgetdc /force` — the combination of subnet overlap + phantom DC (ServerIMC) beat all client-side workarounds. **Rule:** For IMC domain operations where local subnet overlap exists, go onsite. ### Mixed Email Identity (Google + M365) IMC users are split between Google Workspace and Microsoft 365 — different users on different platforms. When configuring a new user, confirm with Leslie which platform their mailbox lives on before setting up Outlook vs. Gmail. ### Stale AD Objects | Object | Last Logon | Status | Action | |---|---|---|---| | IMC2 (computer) | 2023 | Likely decommissioned | Clean up AD object | | IMC-VM (computer) | 2021 | Dead | Clean up AD object | | ServerIMC (DC) | Active (ICMP) | Phantom/broken DC | ntdsutil metadata cleanup or repair | ### GPO Noise - **DistributedCOM 10016** fires every 5 minutes — RuntimeBroker permission noise. Cosmetic. - **Group Policy event 103** fires every 5 minutes — "removal of the assignment of application Syncro from policy Management SW failed". Stale GPO object. Cleanup separately. ### Server 2016 EOL Extended support ends **2027-01-12**. Migration window is finite. The memory pressure / AIM reliability incident is additional evidence to push the migration timeline. Mike wants to scope cost/timeline at next ACG strategy call. --- ## Active Work As of 2026-05-07 (last decision recorded): 1. **[IMMEDIATE] Apply `max server memory` caps on IMC1 SQL instances** — Mike approved 2026-05-07. Howard to implement: SQLEXPRESS 12 GB, WID 512 MB, AIMSQL 256 MB. Reversible (1-second config change, no service restart). Until applied, AIM connection-broken errors will continue recurring. [unverified — confirm applied] 2. **[HIGH] Open ticket for ServerIMC phantom DC investigation** — SRV/A records in DNS claim it's a DC; LDAP/Kerberos refuse connections. Degrades authentication for all domain users. No ticket opened as of 2026-05-06. 3. **[MEDIUM] AIMSQL orphan consolidation** — Mike approved (2026-05-07). Pending: - Locate `AIM.mdf`, `IMC.mdf`, `TestConv61223.mdf` and `.ldf` siblings (not in expected path) - Back up 2023-era DBs before shutdown - Verify no applications reference `IMC1\AIMSQL` (TCP 63116) - Stop and uninstall `MSSQL$AIMSQL` 4. **[MEDIUM] WID instance decision** — Verify AD RMS usage. WSUS confirmed unused. If AD RMS also unused, stop WID to free ~300 MB headroom. Mike awaiting Howard's verification before authorizing stop. 5. **[LOWER] Server 2019 migration scoping** — Three paths (component store repair + in-place; in-place without repair; clean build). Clean build is Mike's recommendation. Scope cost/timeline at next ACG strategy call before 2027-01-12 EOL. 6. **[LOWER] Documentation cleanup:** - Update workstation table in `docs/overview.md` with `DESKTOP-KRHQ5TS` / Manda / AIM USER#=4 - Confirm Manda's full name in AD - Disable SMB1 on IMC1 (`Set-SmbServerConfiguration -EnableSMB1Protocol $false`) - Drop `TestConv61223` DB on AIMSQL (leftover 2023 migration test) — safe per enumeration, but back up `.mdf` first - Clean up stale AD computer objects `IMC2`, `IMC-VM` --- ## History Highlights | Date | By | Event | |---|---|---| | ~2026-Q1 | Mike/Howard | Early engagement: 3 new workstations provisioned at Speedway (hostnames, AIM USER#s TBD in billing log) | | 2026-04-11/12/13 | Mike | IMC1 maintenance: RDS removal blocked (component store corruption 0x80073701), SSH installed, 716 GB freed on E: (backup cleanup), GFS retention automated, AIMsi DBs moved C:→S: SSD | | 2026-04-22 | Howard | Attempted remote domain-join of `DESKTOP-KRHQ5TS` over VPN — abandoned after subnet overlap + phantom DC defeated all workarounds | | 2026-04-28 | Howard | Onsite: `DESKTOP-KRHQ5TS` joined to imc.local, Manda (new GM) AD account created, Outlook/M365 configured, Office activated, AIMsi USER#=4 set. Ticket #32218, 1.5 hrs, prepay 14.0→12.5 hrs. ServerIMC confirmed as active authentication degrader. | | 2026-05-04 | Howard | Onsite (0.5 hrs): Station 2 receipt printer reconnected (re-added from \\imc1); VPN installed on Manda's machine. Ticket #32247. | | 2026-05-05 | Howard | AIM "connection broken" investigation. GuruRMM IMC client/site provisioned, IMC1 enrolled. Diagnosed memory pressure; scheduled AIMSQL restart for 02:30 (wrong instance — superseded next day). | | 2026-05-06 | Howard | Station 1 recurrence 12:14 PM. Full instance enumeration revealed SQLEXPRESS = production Standard (not AIMSQL). Wrong-instance restart task unregistered. Corrected diagnosis in session logs and PROJECT_STATE. Feedback memory created. | | 2026-05-07 | Mike | Decision: approved memory caps (SQLEXPRESS 12 GB, WID 512 MB, AIMSQL 256 MB), AIMSQL consolidation pending backup, Server 2016 migration timeline acknowledged, WSUS confirmed unused. | --- ## Compilation Notes Source material: 5 session logs (2026-04-12 through 2026-05-06) + 1 decision file (2026-05-07) + README + PROJECT_STATE + 10 docs files (most docs/* are blank templates with no client-specific data filled in — network/firewall/vlans/VLAN/DHCP/DNS/RMM/AV/backup/issues docs are all empty templates). Many structured docs (`docs/network/`, `docs/security/`, `docs/cloud/`) are empty templates. The authoritative information sources are `README.md`, `PROJECT_STATE.md`, and the session logs. **Unverified items flagged:** - Whether Howard applied `max server memory` caps after Mike's 2026-05-07 approval - ServerIMC ticket status — ticket was recommended but not confirmed opened - Manda's full name in AD - M365 tenant details (tenant domain, license type, MFA policy) - WID instance AD RMS usage - AIMSQL `.mdf` file locations - Full workstation fleet AIM USER# assignments - ISP, firewall hardware, VLAN/network topology ## Backlinks - [[projects/gururmm]] — IMC1 enrolled as agent `fa99e913-1027-4e33-a928-7695e31068e7`; site IMCMain