azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f2ece8eccf0af2bf03c35b7de4e8d7d4fb3d9941
claudetools/wiki/clients/instrumental-music-center.md
Mike Swanson b583aeed21 wiki: seed Instrumental Music Center + Valley Wide Plastering articles 
instrumental-music-center.md — AIMsi POS on SQL Server 2019 (Standard
under misleading SQLEXPRESS instance name); phantom DC ServerIMC causing
slow logons; GuruRMM enrolled (IMC1 fa99e913); OpenVPN subnet-overlap
hazard; $175/hr prepaid, 12.5 hrs remaining; SQL max server memory fix
approved but unverified applied.

valleywide.md — Valley Wide Plastering; HP DL360 Gen10 VM host + XenServer;
VB6/Access 97 app modernization (130 tables, 791 Crystal Reports, certified
payroll); RDWeb brute-force incident (contained); 11 Yealink phones pending;
iLO requires paramiko (legacy ssh-rsa); $175/hr prepaid, 10 hrs remaining.

wiki/index.md — both clients added to Clients table and Cross-Reference.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-24 19:30:27 -07:00

22 KiB
Raw Blame History

type, name, display_name, last_compiled, compiled_by, sources, backlinks
type name display_name last_compiled compiled_by sources backlinks
client instrumental-music-center Instrumental Music Center 2026-05-24 DESKTOP-0O8A1RL/claude-main
clients/instrumental-music-center/README.md
clients/instrumental-music-center/PROJECT_STATE.md
clients/instrumental-music-center/docs/overview.md
clients/instrumental-music-center/docs/billing-log.md
clients/instrumental-music-center/docs/2026-04-13-ticket-notes.md
clients/instrumental-music-center/docs/network/topology.md
clients/instrumental-music-center/docs/network/vlans.md
clients/instrumental-music-center/docs/network/firewall.md
clients/instrumental-music-center/docs/network/dhcp.md
clients/instrumental-music-center/docs/network/dns.md
clients/instrumental-music-center/docs/cloud/m365.md
clients/instrumental-music-center/docs/cloud/azure.md
clients/instrumental-music-center/docs/rmm/rmm.md
clients/instrumental-music-center/docs/security/antivirus.md
clients/instrumental-music-center/docs/security/backup.md
clients/instrumental-music-center/docs/issues/log.md
clients/instrumental-music-center/docs/servers/server_template.md
clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md
clients/instrumental-music-center/session-logs/2026-04-28-howard-manda-laptop-provision.md
clients/instrumental-music-center/session-logs/2026-05-04-station2-printer-and-manda-vpn.md
clients/instrumental-music-center/session-logs/2026-05-05-howard-aim-connection-broken-investigation.md
clients/instrumental-music-center/session-logs/2026-05-06-howard-imc1-aim-instance-correction.md
clients/instrumental-music-center/decisions/2026-05-07-mike-memory-allocation-approval.md
projects/gururmm

Instrumental Music Center

Music retail and instrument repair shop running AIMsi point-of-sale software on-prem. Single-site as far as documented. Located at 7063 E Speedway Blvd, Tucson AZ 85710. ACG provides managed break-fix / prepaid-block support; primary focus is on the AIMsi SQL server (IMC1) and workstation fleet.

Profile

  • Contract type: Prepaid hour block
  • Billing rate: $175/hr all labor
  • Hours remaining: 12.5 hrs as of 2026-04-28 (after debiting 1.5 hrs for Syncro #32218). Always live-check before billing.
  • Syncro customer ID: 7088508
  • Key contacts:
    • Leslie Stirm — primary contact; leslie@imc-az.com; Syncro contact_id 731730
    • Manda — General Manager (new, replaced Michael Santander as of ~2026-04-28). Full name unconfirmed in AD. [unverified]
    • Michael Santander — former GM; domain account already deactivated.
  • Primary domain: imc.local (on-prem AD)
  • Location: Speedway (7063 E Speedway Blvd, Tucson AZ 85710) — additional locations TBD; only Speedway is documented.
  • Critical software: AIMsi by Tri-Tech (https://www.tritechretail.com/topic/aim) — retail POS and inventory management.

Infrastructure

Servers & Services

Host IP Role OS Notes
IMC1 192.168.0.2 DC (imc.local), DNS, File Server, AIMsi SQL host, RDS host Windows Server 2016 Standard (build 14393.7426) Dell R720, 4 physical cores, 32 GB RAM. GuruRMM agent: fa99e913-1027-4e33-a928-7695e31068e7
ServerIMC 192.168.0.63 Phantom / broken DC Windows Server 2016 Essentials [unverified] [WARNING] Registered as DC in AD DNS (A + SRV records for _ldap._tcp.dc._msdcs.imc.local and _kerberos._tcp.imc.local). Responds to ICMP but TCP/389 (LDAP) and TCP/88 (Kerberos) refuse connections. DC locator round-robins — clients that pick ServerIMC time out. Root cause of intermittent slow logons, GPO failures, and 2026-04-22 remote domain-join failure. Needs ntdsutil metadata cleanup (if demoted ghost) or AD service repair.
IMC2 — Unknown (stale) Windows Server 2016 Essentials Last logon 2023 — likely decommissioned. Clean up AD computer object.
IMC-VM — Unknown (dead) Windows Server 2016 Standard Last logon 2021 — dead. Clean up AD computer object.
Station 1 192.168.0.50 POS workstation Windows [unverified] Hostname IMC-STATION1. Primary workstation for AIM "connection broken" incidents.

IMC1 SQL Instances (CRITICAL — read carefully)

[WARNING] The production AIM database is on IMC1\SQLEXPRESS, NOT IMC1\AIMSQL. The instance name is actively misleading — someone installed SQL Server 2019 Standard under the default SQLEXPRESS instance name and never renamed it. This burned a full day of triage. Always verify SQL roles by active connections (sys.dm_exec_sessions) — never by instance name.

Instance Port Edition (actual) Role Production DB Notes
IMC1\SQLEXPRESS TCP 61151 SQL Server 2019 Standard (misleading name!) PRODUCTION IMCAIM (created 2023-08-21) Service account IMC\AIM. ~9 store workstations + 22 server-local AIM sessions. Do not stop, do not uninstall. ERRORLOG at E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Log\. No max server memory cap (default unlimited).
IMC1\AIMSQL TCP 63116 (dynamic) SQL Server 2019 Express GDR 15.0.2165.1 Orphan (consolidation candidate) None active Service account IMC\IMC1$. Zero established TCP connections. Holds only 2023-era conversion-test DBs (AIM, IMC, TestConv61223). No active backup chain landing here. Shutdown + uninstall approved by Mike pending .mdf backup confirmation.
IMC1\MICROSOFT##WID — Windows Internal Database WSUS / AD RMS — WSUS confirmed NOT in use at IMC. AD RMS status unverified. If AD RMS also unused, instance can be stopped to free ~300 MB. Canary for memory pressure — Event 17890 paging events fire here first when the host is memory-squeezed.

Workstations connected to production IMC1\SQLEXPRESS (verified 2026-05-06):

Hostname IP
IMC-MINI 192.168.0.72
IMC-SVCSTR 192.168.0.55
IMC-LESSONS 192.168.0.62
IMC-STATION2 192.168.0.66
IMC-L1-STATION9 192.168.0.41
DESKTOP-44L80C0 192.168.0.46
DESKTOP-MR3ALTK 192.168.0.59
REPAIRADMIN 192.168.0.48
C2B 192.168.0.4
IMC-STATION1 192.168.0.50

All sessions authenticate as AIMUser1 via .Net SqlClient Data Provider.

IMC1 Disk Layout

Drive Purpose Notes
C: OS, IIS, system DBs 419 GB volume; ~278 GB used after 2026-04-12 cleanup (~66%); was 77% full before. Monitor.
E: SQL backups + installers + Server 2016 media E:\W2016\sources\install.wim is RTM 14393.0. SQL backups at E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\
F: Windows Image Backups —
S: Dedicated SSD (Samsung 850 PRO 256 GB) — AIMsi SQL user DBs User DBs at S:\SQL\Data\. AIM client share \\IMC1\AIM → S:\AIM. System DBs remain on C:.

Email & Identity

  • Mail: IMC uses a mixed Google / Microsoft identity model — different users are on different platforms. Manda is on the M365 side. [full tenant details unverified]
  • M365 tenant details: Not fully documented. Manda's Outlook was configured against an existing M365 mailbox.
  • On-prem AD domain: imc.local
  • MFA status: [unverified]
  • DNS: IMC1 (192.168.0.2) is the authoritative DNS server for imc.local. ServerIMC (192.168.0.63) has ghost A + SRV records as a DC — these are the direct cause of client authentication failures and need cleanup.

Network

  • LAN subnet: 192.168.0.0/24
  • VPN: OpenVPN (.ovpn profile). [WARNING] 192.168.0.0/24 subnet overlap hazard: if technician's home/office LAN is also 192.168.0.0/24 (Howard's home is), OpenVPN routes win for reaching IMC1 but Windows multi-homed DNS races between the two interfaces. DNS negative caching causes domain join / locator failures. If remote LAN overlaps IMC's subnet, go onsite for domain joins. Also: disconnect Tailscale before connecting to IMC OpenVPN — Tailscale's pfsense-2 subnet router advertises 192.168.0.0/24 with lower metric than the VPN, making IMC1 unreachable.
  • Firewall: [unverified — not documented]
  • ISP: [unverified]
  • SMB: SMB1 still enabled on IMC1 — disable as security hygiene when opportunity permits.
  • SMB signing: RequireSecuritySignature = True on server — adds auth overhead.

GuruRMM Enrollment

Field Value
GuruRMM client Instrumental Music Center
GuruRMM client ID 213b62a8-30f4-41dd-9bb3-549341104416
GuruRMM client code IMC
Site IMCMain
Site ID 2c5b65ad-2d5e-47b3-b12b-632e35e08ff6
Site code INNER-BRIDGE-8354
Site enrollment key vault: clients/imc/gururmm-site-main.sops.yaml
First enrolled agent IMC1 (fa99e913-1027-4e33-a928-7695e31068e7)

IMC was enrolled in GuruRMM on 2026-05-05 (Howard, prompted by AIM connection-broken investigation). IMC1 agent was installed by Mike via ScreenConnect. Only IMC1 is enrolled as of last session — workstations not yet enrolled.

Note: When SSH from Howard-Home is blocked by the 192.168.0.0/24 route collision, GuruRMM remote commands are the fallback for running diagnostics on IMC1.

Access

  • SSH: ssh IMC\guru@192.168.0.2 — ed25519 key auth; PowerShell is the default shell. Authorized keys: C:\ProgramData\ssh\administrators_authorized_keys (inheritance off, Administrators + SYSTEM full control).
  • VPN: OpenVPN (.ovpn profile). Disconnect Tailscale first. If home/office LAN is 192.168.0.0/24, remote domain operations will fail — go onsite instead.
  • Domain admin: IMC\guru — also SQL sysadmin on both SQLEXPRESS and AIMSQL (added via single-user recovery 2026-04-12).
  • GuruRMM: IMC1 agent fa99e913-1027-4e33-a928-7695e31068e7 — use for remote commands when SSH is blocked.
  • Vault paths:
    • IMC1 credentials (domain admin, SSH): clients/imc/imc1.sops.yaml
    • GuruRMM site enrollment key: clients/imc/gururmm-site-main.sops.yaml

[WARNING] sa account on AIMSQL: exists and enabled; password unknown. One candidate was tried and failed on 2026-04-12 — no lockout triggered (no lockout policy). If needed for AIMSQL consolidation, use single-user recovery mode (same process used 2026-04-12).

AIMsi / Tri-Tech Critical Notes

Per-machine workstation number (USER#) is mandatory. AIMsi requires a user environment variable USER# (older Tri-Tech convention, still in use at IMC) set on each machine. This is the per-machine workstation identifier for POS polling and licensing.

  • NEVER wipe or reimage a machine without recording its USER# first.
  • When deploying a new machine, assign its USER# per Leslie — she tracks the allocation.
  • Tri-Tech docs: https://www.tritechretail.com/topic/aim

Known USER# assignments:

Machine Hostname USER# Notes
Manda (GM) laptop DESKTOP-KRHQ5TS 4 Assigned per Leslie, 2026-04-28
Other workstations Various TBD Not yet fully documented

Backups

  • Local SQL backups: Nightly at 22:00 to E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\IMCAIM_*.bak
  • Retention script: C:\Scripts\Clean-AimsiBackups.ps1 — GFS policy: 14 dailies + 1st-of-month; 3-newest safety override; logs to C:\Scripts\Logs\aimsi-retention-YYYYMM.log
  • Retention task: IMC AIMsi Backup Retention — daily 23:30, SYSTEM, 1-hour limit
  • Off-site: Cloudberry / MSP360 at C:\ProgramData\Online Backup\. Cloudberry chain confirmed intact before 2026-04-12 deletion run.
    • SQLEXPRESS backup also confirmed landing at C:\ProgramData\Online Backup\MSSQL\IMC1_SQLEXPRESS\
  • Windows Image Backup: on F:
  • AIMSQL orphan: no backup chain. Locate and back up AIM.mdf, IMC.mdf, TestConv61223.mdf and their .ldf siblings before any consolidation — files were not found in expected path under MSSQL15.AIMSQL\MSSQL\DATA or S:\*AIMSQL* during 2026-05-06 search.

Patterns & Known Issues

[WARNING] Phantom DC ServerIMC — Active Authentication Degrader

ServerIMC (192.168.0.63) is registered in DNS as a domain controller (A record + SRV records for _ldap._tcp.dc._msdcs.imc.local and _kerberos._tcp.imc.local) alongside IMC1. It responds to ICMP ping but TCP/389 and TCP/88 refuse connections. The DC locator round-robins between IMC1 and ServerIMC, timing out ~50% of the time.

Effect: Intermittent slow logons, GPO failures, and broken remote domain joins for every domain client at IMC. Was the confirmed root cause of the 2026-04-22 failed remote join of DESKTOP-KRHQ5TS.

Action needed: Open a ticket. Either:

  1. Repair AD services if ServerIMC is a real machine with broken services, or
  2. Run ntdsutil metadata cleanup if it is a ghost from a previously demoted DC.

This was first flagged as "unclear" on 2026-04-13, promoted to confirmed issue 2026-04-28. No ticket has been opened as of 2026-05-06.

AIM "Connection Broken" — Memory Pressure on IMC1

Symptom: Telerik.OpenAccess.RT.sql.SQLException: Connection has been closed / The connection is broken and recovery is not possible — user-facing AIM crash. First seen 2026-05-05 on Station 1 (IMC-STATION1, 192.168.0.50), recurred 2026-05-06 ~12:14 PM.

Root cause: IMC1 is hosting DC services + 6 concurrent RDP users + AIMsi Webservice/Runtime + three SQL instances + QuickBooks Enterprise on 32 GB. Under memory pressure, Windows trims SQL working sets (visible as WID Event 17890 paging events — the canary). The trim reaps idle Telerik OpenAccess TCP pool slots. Telerik has no transient-fault retry, so the next query against a dead pool handle throws the raw stack trace.

SQLEXPRESS has no max server memory cap (default 2,147,483,647 MB). Working set observed at 6.86 GB.

Approved fix (Mike, 2026-05-07): Cap max server memory on each instance:

  • SQLEXPRESS: 12,288 MB (12 GB)
  • MSSQL$MICROSOFT##WID: 512 MB
  • MSSQL$AIMSQL: 256 MB (or consolidate it)

Status as of 2026-05-06: Howard is awaiting go-ahead for implementation. Mike approved on 2026-05-07. Confirm whether Howard has applied the caps — this is the immediate recurrence prevention. [unverified post-2026-05-07]

[WARNING] SQL Instance Name Trap

IMC1\SQLEXPRESS is SQL Server 2019 Standard Edition — someone installed Standard under the default SQLEXPRESS instance name and never renamed it. SERVERPROPERTY('Edition') is the only way to confirm this. The instance name actively misleads.

Never assume an instance is idle, orphan, or Express based on name. Always verify by:

  1. SERVERPROPERTY('Edition') for edition
  2. sys.dm_exec_sessions for active user sessions
  3. Get-NetTCPConnection -OwningProcess for established TCP connections

This trap caused a wrong-instance restart task to be deployed (2026-05-05) that had zero effect on the user-facing problem and was unregistered the next day (2026-05-06). See .claude/memory/feedback_sql_instance_role_by_connection.md.

Component Store Corruption on IMC1 (RDS Removal Blocked)

COMPONENTS registry hive is ~168 MB (normal 30-50 MB), causing 0x80073701 ERROR_SXS_ASSEMBLY_MISSING on any role removal or CU apply-on-boot. ETW manifest for provider GUID {9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0} is malformed — causes CBS_E_INSTALLERS_FAILED → full rollback even when CU staging succeeds.

Effect: Blocks RDS role removal, which was the original reason for the 2026-04-12 engagement. Also means CU KB5075999 cannot be applied cleanly.

Server is otherwise healthy — AIMsi production is running. This is a structural impediment to the Server 2019 migration. Three paths considered (see History Highlights).

Remote Domain Join Over OpenVPN — Don't Do It

If the technician's local LAN subnet overlaps IMC's 192.168.0.0/24, remote domain joins over OpenVPN will fail reliably:

  • OpenVPN pushed routes win for TCP, but Windows multi-homed DNS races between LAN DNS and VPN DNS (both respond to imc.local queries; LAN returns NXDOMAIN faster; Windows caches the negative answer).
  • Even with NRPT rules, hosts file entries, -Server <IP> on Add-Computer, and nltest /dsgetdc /force — the combination of subnet overlap + phantom DC (ServerIMC) beat all client-side workarounds.

Rule: For IMC domain operations where local subnet overlap exists, go onsite.

Mixed Email Identity (Google + M365)

IMC users are split between Google Workspace and Microsoft 365 — different users on different platforms. When configuring a new user, confirm with Leslie which platform their mailbox lives on before setting up Outlook vs. Gmail.

Stale AD Objects

Object Last Logon Status Action
IMC2 (computer) 2023 Likely decommissioned Clean up AD object
IMC-VM (computer) 2021 Dead Clean up AD object
ServerIMC (DC) Active (ICMP) Phantom/broken DC ntdsutil metadata cleanup or repair

GPO Noise

  • DistributedCOM 10016 fires every 5 minutes — RuntimeBroker permission noise. Cosmetic.
  • Group Policy event 103 fires every 5 minutes — "removal of the assignment of application Syncro from policy Management SW failed". Stale GPO object. Cleanup separately.

Server 2016 EOL

Extended support ends 2027-01-12. Migration window is finite. The memory pressure / AIM reliability incident is additional evidence to push the migration timeline. Mike wants to scope cost/timeline at next ACG strategy call.

Active Work

As of 2026-05-07 (last decision recorded):

  1. [IMMEDIATE] Apply max server memory caps on IMC1 SQL instances — Mike approved 2026-05-07. Howard to implement: SQLEXPRESS 12 GB, WID 512 MB, AIMSQL 256 MB. Reversible (1-second config change, no service restart). Until applied, AIM connection-broken errors will continue recurring. [unverified — confirm applied]

  2. [HIGH] Open ticket for ServerIMC phantom DC investigation — SRV/A records in DNS claim it's a DC; LDAP/Kerberos refuse connections. Degrades authentication for all domain users. No ticket opened as of 2026-05-06.

  3. [MEDIUM] AIMSQL orphan consolidation — Mike approved (2026-05-07). Pending:

    • Locate AIM.mdf, IMC.mdf, TestConv61223.mdf and .ldf siblings (not in expected path)
    • Back up 2023-era DBs before shutdown
    • Verify no applications reference IMC1\AIMSQL (TCP 63116)
    • Stop and uninstall MSSQL$AIMSQL

  4. [MEDIUM] WID instance decision — Verify AD RMS usage. WSUS confirmed unused. If AD RMS also unused, stop WID to free ~300 MB headroom. Mike awaiting Howard's verification before authorizing stop.

  5. [LOWER] Server 2019 migration scoping — Three paths (component store repair + in-place; in-place without repair; clean build). Clean build is Mike's recommendation. Scope cost/timeline at next ACG strategy call before 2027-01-12 EOL.

  6. [LOWER] Documentation cleanup:

    • Update workstation table in docs/overview.md with DESKTOP-KRHQ5TS / Manda / AIM USER#=4
    • Confirm Manda's full name in AD
    • Disable SMB1 on IMC1 (Set-SmbServerConfiguration -EnableSMB1Protocol $false)
    • Drop TestConv61223 DB on AIMSQL (leftover 2023 migration test) — safe per enumeration, but back up .mdf first
    • Clean up stale AD computer objects IMC2, IMC-VM

History Highlights

Date By Event
~2026-Q1 Mike/Howard Early engagement: 3 new workstations provisioned at Speedway (hostnames, AIM USER#s TBD in billing log)
2026-04-11/12/13 Mike IMC1 maintenance: RDS removal blocked (component store corruption 0x80073701), SSH installed, 716 GB freed on E: (backup cleanup), GFS retention automated, AIMsi DBs moved C:→S: SSD
2026-04-22 Howard Attempted remote domain-join of DESKTOP-KRHQ5TS over VPN — abandoned after subnet overlap + phantom DC defeated all workarounds
2026-04-28 Howard Onsite: DESKTOP-KRHQ5TS joined to imc.local, Manda (new GM) AD account created, Outlook/M365 configured, Office activated, AIMsi USER#=4 set. Ticket #32218, 1.5 hrs, prepay 14.0→12.5 hrs. ServerIMC confirmed as active authentication degrader.
2026-05-04 Howard Onsite (0.5 hrs): Station 2 receipt printer reconnected (re-added from \imc1); VPN installed on Manda's machine. Ticket #32247.
2026-05-05 Howard AIM "connection broken" investigation. GuruRMM IMC client/site provisioned, IMC1 enrolled. Diagnosed memory pressure; scheduled AIMSQL restart for 02:30 (wrong instance — superseded next day).
2026-05-06 Howard Station 1 recurrence 12:14 PM. Full instance enumeration revealed SQLEXPRESS = production Standard (not AIMSQL). Wrong-instance restart task unregistered. Corrected diagnosis in session logs and PROJECT_STATE. Feedback memory created.
2026-05-07 Mike Decision: approved memory caps (SQLEXPRESS 12 GB, WID 512 MB, AIMSQL 256 MB), AIMSQL consolidation pending backup, Server 2016 migration timeline acknowledged, WSUS confirmed unused.

Compilation Notes

Source material: 5 session logs (2026-04-12 through 2026-05-06) + 1 decision file (2026-05-07) + README + PROJECT_STATE + 10 docs files (most docs/* are blank templates with no client-specific data filled in — network/firewall/vlans/VLAN/DHCP/DNS/RMM/AV/backup/issues docs are all empty templates).

Many structured docs (docs/network/, docs/security/, docs/cloud/) are empty templates. The authoritative information sources are README.md, PROJECT_STATE.md, and the session logs.

Unverified items flagged:

  • Whether Howard applied max server memory caps after Mike's 2026-05-07 approval
  • ServerIMC ticket status — ticket was recommended but not confirmed opened
  • Manda's full name in AD
  • M365 tenant details (tenant domain, license type, MFA policy)
  • WID instance AD RMS usage
  • AIMSQL .mdf file locations
  • Full workstation fleet AIM USER# assignments
  • ISP, firewall hardware, VLAN/network topology

Backlinks

  • projects/gururmm — IMC1 enrolled as agent fa99e913-1027-4e33-a928-7695e31068e7; site IMCMain
Reference in New Issue View Git Blame Copy Permalink