--- type: client name: instrumental-music-center display_name: Instrumental Music Center last_compiled: 2026-07-10 compiled_by: HOWARD-HOME/claude-main sources: - projects/gps-rmm-audit/session-logs/2026-07/2026-07-07-howard-gps-master-status-wikis-onsite-kit.md - clients/instrumental-music-center/README.md - clients/instrumental-music-center/PROJECT_STATE.md - clients/instrumental-music-center/docs/overview.md - clients/instrumental-music-center/docs/billing-log.md - clients/instrumental-music-center/docs/2026-04-13-ticket-notes.md - clients/instrumental-music-center/docs/network/topology.md - clients/instrumental-music-center/docs/network/vlans.md - clients/instrumental-music-center/docs/network/firewall.md - clients/instrumental-music-center/docs/network/dhcp.md - clients/instrumental-music-center/docs/network/dns.md - clients/instrumental-music-center/docs/cloud/m365.md - clients/instrumental-music-center/docs/cloud/azure.md - clients/instrumental-music-center/docs/rmm/rmm.md - clients/instrumental-music-center/docs/security/antivirus.md - clients/instrumental-music-center/docs/security/backup.md - clients/instrumental-music-center/docs/issues/log.md - clients/instrumental-music-center/docs/servers/server_template.md - clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md - clients/instrumental-music-center/session-logs/2026-04-28-howard-manda-laptop-provision.md - clients/instrumental-music-center/session-logs/2026-05-04-station2-printer-and-manda-vpn.md - clients/instrumental-music-center/session-logs/2026-05-05-howard-aim-connection-broken-investigation.md - clients/instrumental-music-center/session-logs/2026-05-06-howard-imc1-aim-instance-correction.md - clients/instrumental-music-center/decisions/2026-05-07-mike-memory-allocation-approval.md - clients/instrumental-music-center/session-logs/2026-07/2026-07-08-howard-imc-edr-rollout-sql-caps.md backlinks: - projects/gururmm --- # Instrumental Music Center Music retail and instrument repair shop running AIMsi point-of-sale software on-prem. **Two locations:** the main store on Speedway (7063 E Speedway Blvd, Tucson AZ 85710) and a second location, **MOO** (6300 E El Dorado Cir, Tucson AZ 85715). ACG provides managed break-fix / prepaid-block support; primary focus is on the AIMsi SQL server (IMC1) and workstation fleet. In GuruRMM the two locations are the **Main** and **MOO** sites (see GuruRMM Enrollment). --- ## Profile - **Contract type:** Prepaid hour block - **Billing rate:** $175/hr all labor - **Hours remaining:** 10.0 hrs as of 2026-07-10 (live Syncro). Always live-check before billing. (0 open Syncro tickets as of 2026-07-10.) - **Syncro customer ID:** 7088508 - **Key contacts:** - **Leslie Stirm** — primary contact; leslie@imc-az.com; Syncro contact_id 731730 - **Manda** — General Manager (new, replaced Michael Santander as of ~2026-04-28). Full name unconfirmed in AD. [unverified] - **Michael Santander** — former GM; domain account already deactivated. - **Primary domain:** imc.local (on-prem AD) - **Locations (2):** **Main** — Speedway (7063 E Speedway Blvd, Tucson AZ 85710); **MOO** — 6300 E El Dorado Cir, Tucson AZ 85715. Mapped as separate GuruRMM sites 2026-07-07. - **Critical software:** AIMsi by Tri-Tech (https://www.tritechretail.com/topic/aim) — retail POS and inventory management. --- ## Infrastructure ### Servers & Services | Host | IP | Role | OS | Notes | |---|---|---|---|---| | IMC1 | 192.168.0.2 | DC (imc.local), DNS, File Server, AIMsi SQL host, RDS host | Windows Server 2016 Standard (build 14393.7426) | Dell R720, 4 physical cores, 32 GB RAM. GuruRMM agent: `fa99e913-1027-4e33-a928-7695e31068e7` | | ServerIMC | 192.168.0.63 | Phantom / broken DC | Windows Server 2016 Essentials [unverified] | **[WARNING] Registered as DC in AD DNS (A + SRV records for `_ldap._tcp.dc._msdcs.imc.local` and `_kerberos._tcp.imc.local`). Responds to ICMP but TCP/389 (LDAP) and TCP/88 (Kerberos) refuse connections. DC locator round-robins — clients that pick ServerIMC time out. Root cause of intermittent slow logons, GPO failures, and 2026-04-22 remote domain-join failure. Needs `ntdsutil` metadata cleanup (if demoted ghost) or AD service repair.** | | IMC2 | — | Unknown (stale) | Windows Server 2016 Essentials | Last logon 2023 — likely decommissioned. Clean up AD computer object. | | IMC-VM | — | Unknown (dead) | Windows Server 2016 Standard | Last logon 2021 — dead. Clean up AD computer object. | | Station 1 | 192.168.0.50 | POS workstation | Windows [unverified] | Hostname `IMC-STATION1`. Primary workstation for AIM "connection broken" incidents. | #### IMC1 SQL Instances (CRITICAL — read carefully) **[WARNING] The production AIM database is on `IMC1\SQLEXPRESS`, NOT `IMC1\AIMSQL`. The instance name is actively misleading — someone installed SQL Server 2019 Standard under the default `SQLEXPRESS` instance name and never renamed it. This burned a full day of triage. Always verify SQL roles by active connections (`sys.dm_exec_sessions`) — never by instance name.** | Instance | Port | Edition (actual) | Role | Production DB | Notes | |---|---|---|---|---|---| | `IMC1\SQLEXPRESS` | TCP 61151 | **SQL Server 2019 Standard** (misleading name!) | **PRODUCTION** | `IMCAIM` (created 2023-08-21) | Service account `IMC\AIM`. ~9 store workstations + 22 server-local AIM sessions. **Do not stop, do not uninstall.** ERRORLOG at `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Log\`. No `max server memory` cap (default unlimited). | | `IMC1\AIMSQL` | TCP 63116 (dynamic) | SQL Server 2019 Express GDR 15.0.2165.1 | **Orphan** (consolidation candidate) | None active | Service account `IMC\IMC1$`. Zero established TCP connections. Holds only 2023-era conversion-test DBs (`AIM`, `IMC`, `TestConv61223`). No active backup chain landing here. Shutdown + uninstall approved by Mike pending `.mdf` backup confirmation. | | `IMC1\MICROSOFT##WID` | — | Windows Internal Database | WSUS / AD RMS | — | WSUS confirmed NOT in use at IMC. AD RMS status unverified. If AD RMS also unused, instance can be stopped to free ~300 MB. **Canary for memory pressure** — Event 17890 paging events fire here first when the host is memory-squeezed. | **Workstations connected to production `IMC1\SQLEXPRESS` (verified 2026-05-06):** | Hostname | IP | |---|---| | IMC-MINI | 192.168.0.72 | | IMC-SVCSTR | 192.168.0.55 | | IMC-LESSONS | 192.168.0.62 | | IMC-STATION2 | 192.168.0.66 | | IMC-L1-STATION9 | 192.168.0.41 | | DESKTOP-44L80C0 | 192.168.0.46 | | DESKTOP-MR3ALTK | 192.168.0.59 | | REPAIRADMIN | 192.168.0.48 | | C2B | 192.168.0.4 | | IMC-STATION1 | 192.168.0.50 | All sessions authenticate as `AIMUser1` via `.Net SqlClient Data Provider`. #### IMC1 Disk Layout | Drive | Purpose | Notes | |---|---|---| | C: | OS, IIS, system DBs | 419 GB volume; ~278 GB used after 2026-04-12 cleanup (~66%); was 77% full before. Monitor. | | E: | SQL backups + installers + Server 2016 media | `E:\W2016\sources\install.wim` is RTM 14393.0. SQL backups at `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\` | | F: | Windows Image Backups | — | | S: | Dedicated SSD (Samsung 850 PRO 256 GB) — AIMsi SQL user DBs | User DBs at `S:\SQL\Data\`. AIM client share `\\IMC1\AIM` → `S:\AIM`. System DBs remain on C:. | ### Email & Identity - **Mail:** IMC uses a **mixed Google / Microsoft identity model** — different users are on different platforms. Manda is on the M365 side. [full tenant details unverified] - **M365 tenant details:** Not fully documented. Manda's Outlook was configured against an existing M365 mailbox. - **On-prem AD domain:** `imc.local` - **MFA status:** [unverified] - **DNS:** IMC1 (192.168.0.2) is the authoritative DNS server for imc.local. ServerIMC (192.168.0.63) has ghost A + SRV records as a DC — these are the direct cause of client authentication failures and need cleanup. ### Network - **LAN subnet:** 192.168.0.0/24 - **VPN:** OpenVPN (.ovpn profile). **[WARNING] 192.168.0.0/24 subnet overlap hazard:** if technician's home/office LAN is also 192.168.0.0/24 (Howard's home is), OpenVPN routes win for reaching IMC1 but Windows multi-homed DNS races between the two interfaces. DNS negative caching causes domain join / locator failures. **If remote LAN overlaps IMC's subnet, go onsite for domain joins.** Also: disconnect Tailscale before connecting to IMC OpenVPN — Tailscale's `pfsense-2` subnet router advertises 192.168.0.0/24 with lower metric than the VPN, making IMC1 unreachable. - **Firewall:** [unverified — not documented] - **ISP:** [unverified] - **SMB:** SMB1 still enabled on IMC1 — disable as security hygiene when opportunity permits. - **SMB signing:** `RequireSecuritySignature = True` on server — adds auth overhead. --- ## GuruRMM Enrollment GuruRMM client **Instrumental Music Center** (`213b62a8-30f4-41dd-9bb3-549341104416`, code `IMC`). **Two sites, one per physical location** (created/split 2026-07-07): | Site | Site ID | Site code | Enrollment key (vault) | |---|---|---|---| | **Main** (Speedway; renamed from `IMCMain`) | `2c5b65ad-2d5e-47b3-b12b-632e35e08ff6` | `INNER-BRIDGE-8354` | `clients/imc/gururmm-site-main.sops.yaml` | | **MOO** (second location) | `f9e4a897-708b-43c5-ab7d-a8b05decdc3e` | `SILVER-BRIDGE-5118` | `clients/imc/gururmm-site-moo.sops.yaml` | **Agent placement (as of 2026-07-07):** - **MOO (3):** `BigMom`, `DESKTOP-GHG12G3`, `IMC-LUIS` - **Main (14):** `IMC1`, `C2B`, `IMC-STATION1`, `IMC-STATION2`, `IMC-L1-STATION9`, `DESKTOP-44L80C0` (a.k.a. "Office 2"), `DESKTOP-MR3ALTK` (a.k.a. "Office 1"), `IMC-Mini`, `IMC-Lessons`, `IMC-SvcStr`, `IMC-M-EdServices1`, `RepairAdmin`, `LAPTOP-DCHQ3F92`, `LAPTOP-UBTI0IE3` IMC was first enrolled in GuruRMM 2026-05-05 (IMC1 only, via ScreenConnect). The rest of the fleet was enrolled during the GPS -> GuruRMM audit (bulk SC push). On 2026-07-07 the client was split into the Main + MOO sites, agents re-homed accordingly, and BigMom (previously SC-only) enrolled into MOO. ScreenConnect metadata matches: Company = Instrumental Music Center, Site = Main/MOO, with location tags (Front for the front-counter stations; Office 1/Office 2 for the office desktops), Department kept as "Sales". **Note:** When SSH from Howard-Home is blocked by the 192.168.0.0/24 route collision, GuruRMM remote commands are the fallback for running diagnostics on IMC1. ### 2026-07-07 fleet reconciliation (RMM / AV / backup / Syncro assets) Cross-referenced GuruRMM vs ScreenConnect vs Datto EDR vs MSP360 vs Syncro assets for IMC. **Site rule:** Main = public IP `72.194.57.2` (Speedway); MOO = `68.107.177.170`; anything else defaults to Main. ScreenConnect CP1 = `IMC - Instrumental Music Center`, CP2 = Site (SC truncates Name to 15 chars, e.g. `IMC-M-EDSERVICE` == RMM `IMC-M-EdServices1`). **GuruRMM enrollments added/queued 2026-07-07 (all Main unless noted):** - **Enrolled + verified:** `DESKTOP-NFU17AJ` (agent count 17 -> 18). - **Queued via ScreenConnect** (single self-contained `iex (irm .../install/INNER-BRIDGE-8354/windows)` in the session's one command slot; runs on reconnect): `DESKTOP-KRHQ5TS` (Manda), `SNAPPERLANDHOME` (Sara Sherer), `LAPTOP-PNVA9G51`, `LESLIE2019LAPTO` (Leslie's events laptop). ## Security — AntiVirus / EDR **IMC's AV/EDR is Datto AV/EDR (Infocyte), NOT Bitdefender.** Datto org **Instrumental Music Center** `00b6b6e9-35ea-47cd-9413-088776434514`, single scan-target group **IMC - Speedway (Loc1)** `c56026f0-b174-4f84-9dc4-cf8cffa49b74`, enrollment **RegKey `instrume01`** (all IMC endpoints — incl. MOO — land in this one org/group). Deploy one-liner: `Install-EDR -URL "https://azcomp4587.infocyte.com" -RegKey instrume01` (push via `/rmm`, or queue via SC send-command for an offline box). - **Bitdefender is being retired from IMC.** Only one IMC endpoint was ever on Bitdefender GravityZone — `REPAIRADMIN` (BD company `5e77afc47357486d522f1a44`, BEST agent, anti-tamper ON). **BD has NO uninstall API** (GravityZone console/local-tool only); policy has a blank uninstall password (`shouldAskForPassword: false`), so removal = push BD's local uninstall tool via `/rmm`. **Swap order: install + verify Datto EDR FIRST, then pull Bitdefender** (no unprotected window). - **EDR deployed 2026-07-07:** `BigMom` (MOO) + `DESKTOP-NFU17AJ` (Main) — installed, exit 0, registered (Datto org 16 -> 18). **Queued via SC** (offline): `IMC-LUIS` (MOO), `IMC-M-EdServices1` (Main), `REPAIRADMIN` (EDR then BD removal). - **Still to cover:** the 4 SC-queued RMM enrollments above get EDR *after* they land in GuruRMM (their one SC slot currently holds the RMM installer). - **3 dead Datto agents** (`desktop-urv3ugr`, `office1`, `purchasingcomp`) = retire-list machines; clear when their Syncro assets are archived. ## Billing (Syncro) — AV yes, Backup unbilled Syncro customer **7088508**; one monthly recurring schedule **#365662** ("GPS/AV/365/Monthly services - Leslie Stirm", $1,590.02/mo). Relevant lines: - **AntiVirus: BILLED** — `GPS AntiVirus Add-on` 20 x $5 = **$100/mo** (20 seats). `GPS basic` also bundles AV in its description — possible partial double-charge, worth a sanity check. - **Backup: NOT billed** — no MSP360/Cloudberry/backup line on the invoice, and `GPS basic`'s inclusion list does not cover backup. **IMC1 IS backing up (unbilled) — revenue gap.** - Other lines: `GPS basic` 18 x $20, `GPS Pro Server` 2 x $80, Pax8 M365 (5), Hardware-Mgmt fees, PacketDial VoIP. - **AV seat reconciliation:** 20 billed vs ~14 actually protected (13 live Datto + 1 BD) at audit time; the 2026-07-07 EDR rollout closes the gap by covering the previously-unprotected live machines (`BigMom`, `IMC-LUIS`, `IMC-M-EdServices1`, `DESKTOP-NFU17AJ`, + REPAIRADMIN swap). **Syncro asset retirement (offline > 40 days):** Syncro "retire" = **Archive** — **UI-only, no REST API** (never `DELETE` — loses history/ticket links). 12 assets qualify (archive via Assets tab -> Bulk Actions -> Archive): `PHIL2021LAPTOP` (5345521), `IMC-EVENTS` (5033192), `DESKTOP-JQ0D38J` (10333758), `DESKTOP-URV3UGR` (2486998), `PURCHASINGCOMP` (2967905), `PHIL` (3100580), `IMC-L1-GRAPHICS` (2451117), `DNW-L8-1..4` (3151232/234/242/229), `DNW-L24` (3150899). Kept (< 40d): `LAPTOP-PNVA9G51`, `DESKTOP-KRHQ5TS`, `IMC-PRINTSERVER`. --- ## Access - **SSH:** `ssh IMC\guru@192.168.0.2` — ed25519 key auth; PowerShell is the default shell. Authorized keys: `C:\ProgramData\ssh\administrators_authorized_keys` (inheritance off, Administrators + SYSTEM full control). - **VPN:** OpenVPN (.ovpn profile). Disconnect Tailscale first. If home/office LAN is 192.168.0.0/24, remote domain operations will fail — go onsite instead. - **Domain admin:** `IMC\guru` — also SQL sysadmin on both SQLEXPRESS and AIMSQL (added via single-user recovery 2026-04-12). - **GuruRMM:** IMC1 agent `fa99e913-1027-4e33-a928-7695e31068e7` — use for remote commands when SSH is blocked. - **Vault paths:** - IMC1 credentials (domain admin, SSH): `clients/imc/imc1.sops.yaml` - GuruRMM site enrollment key: `clients/imc/gururmm-site-main.sops.yaml` **[WARNING] `sa` account on AIMSQL:** exists and enabled; password unknown. One candidate was tried and failed on 2026-04-12 — no lockout triggered (no lockout policy). If needed for AIMSQL consolidation, use single-user recovery mode (same process used 2026-04-12). --- ## AIMsi / Tri-Tech Critical Notes **Per-machine workstation number (`USER#`) is mandatory.** AIMsi requires a user environment variable `USER#` (older Tri-Tech convention, still in use at IMC) set on each machine. This is the per-machine workstation identifier for POS polling and licensing. - **NEVER wipe or reimage a machine without recording its `USER#` first.** - **When deploying a new machine, assign its `USER#` per Leslie** — she tracks the allocation. - Tri-Tech docs: https://www.tritechretail.com/topic/aim **Known `USER#` assignments:** | Machine | Hostname | USER# | Notes | |---|---|---|---| | Manda (GM) laptop | DESKTOP-KRHQ5TS | 4 | Assigned per Leslie, 2026-04-28 | | Other workstations | Various | TBD | Not yet fully documented | --- ## Backups - **Local SQL backups:** Nightly at 22:00 to `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\IMCAIM_*.bak` - **Retention script:** `C:\Scripts\Clean-AimsiBackups.ps1` — GFS policy: 14 dailies + 1st-of-month; 3-newest safety override; logs to `C:\Scripts\Logs\aimsi-retention-YYYYMM.log` - **Retention task:** `IMC AIMsi Backup Retention` — daily 23:30, SYSTEM, 1-hour limit - **Off-site:** Cloudberry / MSP360 at `C:\ProgramData\Online Backup\`. Cloudberry chain confirmed intact before 2026-04-12 deletion run. - SQLEXPRESS backup also confirmed landing at `C:\ProgramData\Online Backup\MSSQL\IMC1_SQLEXPRESS\` - **MSP360 live status (verified 2026-07-07, company "Instrumental Music"):** IMC1 has 3 plans — `Amisi Server - Files` **Success, ran daily** (last 2026-07-07 07:00, ~3.8 GB); `Backup MS SQL Server` **ran daily but "completed with warnings"** (last 2026-07-07 21:00) — warning `LsnDoNotMatchBeforeDiff` forces a full backup of `IMCAIM` because the local nightly SQL job (22:00) and MSP360's diff chain compete for the log chain (align the two jobs to stop the churn); `Amisi Server - Images` **stale — last ran 2024-10-24** (~20 mo; decide keep-or-drop). **Backup is live but NOT billed** — see Billing. [Backup remediation tracked in a separate plan.] - **Windows Image Backup:** on F: - **AIMSQL orphan:** no backup chain. Locate and back up `AIM.mdf`, `IMC.mdf`, `TestConv61223.mdf` and their `.ldf` siblings before any consolidation — files were not found in expected path under `MSSQL15.AIMSQL\MSSQL\DATA` or `S:\*AIMSQL*` during 2026-05-06 search. --- ## Patterns & Known Issues ### [WARNING] Phantom DC `ServerIMC` — Active Authentication Degrader `ServerIMC` (192.168.0.63) is registered in DNS as a domain controller (A record + SRV records for `_ldap._tcp.dc._msdcs.imc.local` and `_kerberos._tcp.imc.local`) alongside IMC1. It responds to ICMP ping but TCP/389 and TCP/88 refuse connections. The DC locator round-robins between IMC1 and ServerIMC, timing out ~50% of the time. **Effect:** Intermittent slow logons, GPO failures, and broken remote domain joins for every domain client at IMC. Was the confirmed root cause of the 2026-04-22 failed remote join of `DESKTOP-KRHQ5TS`. **Action needed:** Open a ticket. Either: 1. Repair AD services if `ServerIMC` is a real machine with broken services, or 2. Run `ntdsutil` metadata cleanup if it is a ghost from a previously demoted DC. This was first flagged as "unclear" on 2026-04-13, promoted to confirmed issue 2026-04-28. No ticket has been opened as of 2026-05-06. ### AIM "Connection Broken" — Memory Pressure on IMC1 **Symptom:** `Telerik.OpenAccess.RT.sql.SQLException: Connection has been closed / The connection is broken and recovery is not possible` — user-facing AIM crash. First seen 2026-05-05 on Station 1 (IMC-STATION1, 192.168.0.50), recurred 2026-05-06 ~12:14 PM. **Root cause:** IMC1 is hosting DC services + 6 concurrent RDP users + AIMsi Webservice/Runtime + three SQL instances + QuickBooks Enterprise on 32 GB. Under memory pressure, Windows trims SQL working sets (visible as WID Event 17890 paging events — the canary). The trim reaps idle Telerik OpenAccess TCP pool slots. Telerik has no transient-fault retry, so the next query against a dead pool handle throws the raw stack trace. **SQLEXPRESS has no `max server memory` cap** (default 2,147,483,647 MB). Working set observed at 6.86 GB. **Approved fix (Mike, 2026-05-07):** Cap `max server memory` on each instance: - `SQLEXPRESS`: 12,288 MB (12 GB) - `MSSQL$MICROSOFT##WID`: 512 MB - `MSSQL$AIMSQL`: 256 MB (or consolidate it) **Status: APPLIED 2026-07-07 (Howard).** Verified live on IMC1 via GuruRMM — `SQLEXPRESS` = **12288 MB**, `AIMSQL` = **256 MB**, `MSSQL$MICROSOFT##WID` = **512 MB** (all read back correct). Before this, all three were at the SQL default `2147483647 MB` (uncapped) — i.e. the caps had never been applied despite Mike's 2026-05-07 approval. Applied via `sp_configure` + `RECONFIGURE`, no service restart. WID set as SYSTEM; SQLEXPRESS + AIMSQL required `IMC\guru` (SQL sysadmin) via token impersonation because the RMM agent's `NT AUTHORITY\SYSTEM` is NOT sysadmin on those two instances. **Rollback:** set each back to `2147483647` (reversible, no restart). **[WATCH — through ~2026-07-11] Probable-cause flag:** if anyone at IMC reports AIM slowness, "connection broken", or any SQL/app performance issue in the next few days, **investigate this memory-cap change FIRST** — it is the most recent production change on IMC1. Most likely suspect: the `SQLEXPRESS` 12 GB cap if the working set legitimately needs more (it was ~6.9 GB at audit, so 12 GB should be ample). Quick revert = raise/remove the cap (`max server memory` back to `2147483647`), takes effect immediately. Confirm-or-clear this flag after a few clean days. **Security note:** applying to SQLEXPRESS/AIMSQL required `IMC\guru`'s password to transit one GuruRMM command (base64 in `command_text`, cmd `378fffe7`, 2026-07-07) — recoverable in the RMM DB. Rotate `IMC\guru` if zero residual exposure is wanted. ### [WARNING] SQL Instance Name Trap **`IMC1\SQLEXPRESS` is SQL Server 2019 Standard Edition** — someone installed Standard under the default `SQLEXPRESS` instance name and never renamed it. `SERVERPROPERTY('Edition')` is the only way to confirm this. The instance name actively misleads. **Never assume an instance is idle, orphan, or Express based on name.** Always verify by: 1. `SERVERPROPERTY('Edition')` for edition 2. `sys.dm_exec_sessions` for active user sessions 3. `Get-NetTCPConnection -OwningProcess` for established TCP connections This trap caused a wrong-instance restart task to be deployed (2026-05-05) that had zero effect on the user-facing problem and was unregistered the next day (2026-05-06). See `.claude/memory/feedback_sql_instance_role_by_connection.md`. ### Component Store Corruption on IMC1 (RDS Removal Blocked) `COMPONENTS` registry hive is ~168 MB (normal 30-50 MB), causing `0x80073701 ERROR_SXS_ASSEMBLY_MISSING` on any role removal or CU apply-on-boot. ETW manifest for provider GUID `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}` is malformed — causes `CBS_E_INSTALLERS_FAILED` → full rollback even when CU staging succeeds. **Effect:** Blocks RDS role removal, which was the original reason for the 2026-04-12 engagement. Also means CU KB5075999 cannot be applied cleanly. **Server is otherwise healthy** — AIMsi production is running. This is a structural impediment to the Server 2019 migration. Three paths considered (see History Highlights). ### Remote Domain Join Over OpenVPN — Don't Do It If the technician's local LAN subnet overlaps IMC's 192.168.0.0/24, remote domain joins over OpenVPN will fail reliably: - OpenVPN pushed routes win for TCP, but Windows multi-homed DNS races between LAN DNS and VPN DNS (both respond to `imc.local` queries; LAN returns NXDOMAIN faster; Windows caches the negative answer). - Even with NRPT rules, hosts file entries, `-Server ` on Add-Computer, and `nltest /dsgetdc /force` — the combination of subnet overlap + phantom DC (ServerIMC) beat all client-side workarounds. **Rule:** For IMC domain operations where local subnet overlap exists, go onsite. ### Mixed Email Identity (Google + M365) IMC users are split between Google Workspace and Microsoft 365 — different users on different platforms. When configuring a new user, confirm with Leslie which platform their mailbox lives on before setting up Outlook vs. Gmail. ### Stale AD Objects | Object | Last Logon | Status | Action | |---|---|---|---| | IMC2 (computer) | 2023 | Likely decommissioned | Clean up AD object | | IMC-VM (computer) | 2021 | Dead | Clean up AD object | | ServerIMC (DC) | Active (ICMP) | Phantom/broken DC | ntdsutil metadata cleanup or repair | ### GPO Noise - **DistributedCOM 10016** fires every 5 minutes — RuntimeBroker permission noise. Cosmetic. - **Group Policy event 103** fires every 5 minutes — "removal of the assignment of application Syncro from policy Management SW failed". Stale GPO object. Cleanup separately. ### Server 2016 EOL Extended support ends **2027-01-12**. Migration window is finite. The memory pressure / AIM reliability incident is additional evidence to push the migration timeline. Mike wants to scope cost/timeline at next ACG strategy call. --- ## Active Work As of 2026-07-07 (fleet reconciliation): 1. **[PENDING] REPAIRADMIN Bitdefender -> Datto EDR swap** — EDR install queued via SC; when the box next comes online, verify EDR live, then push BD's local uninstall tool via `/rmm` (blank uninstall password) and delete the stale BD console entry. 2. **[PENDING] EDR on the SC-queued RMM enrollments** — after `DESKTOP-KRHQ5TS` (Manda), `SNAPPERLANDHOME` (Sara), `LAPTOP-PNVA9G51`, `LESLIE2019LAPTO` land in GuruRMM, push Datto EDR to each (their one SC slot currently holds the RMM installer). 3. **[PENDING - user] Archive 12 offline-> 40d Syncro assets** — UI-only (Assets -> Bulk Actions -> Archive); list in Billing section above. Then remove the 3 matching dead Datto agents. 4. **[SEPARATE PLAN] Backup billing + SQL diff-chain churn** — IMC1 backs up but isn't billed; `Amisi Server - Images` plan stale since 2024-10; SQL diff chain competes with the local 22:00 job. Being handled in a separate backup plan. As of 2026-05-07 (last decision recorded): 1. **[DONE 2026-07-07] Apply `max server memory` caps on IMC1 SQL instances** — APPLIED + verified: SQLEXPRESS 12 GB, AIMSQL 256 MB, WID 512 MB (were all uncapped/default). See the "AIM Connection Broken — Memory Pressure" section for the WATCH/probable-cause flag through ~2026-07-11 and rollback. 2. **[HIGH] Open ticket for ServerIMC phantom DC investigation** — SRV/A records in DNS claim it's a DC; LDAP/Kerberos refuse connections. Degrades authentication for all domain users. No ticket opened as of 2026-05-06. 3. **[MEDIUM] AIMSQL orphan consolidation** — Mike approved (2026-05-07). Pending: - Locate `AIM.mdf`, `IMC.mdf`, `TestConv61223.mdf` and `.ldf` siblings (not in expected path) - Back up 2023-era DBs before shutdown - Verify no applications reference `IMC1\AIMSQL` (TCP 63116) - Stop and uninstall `MSSQL$AIMSQL` 4. **[MEDIUM] WID instance decision** — Verify AD RMS usage. WSUS confirmed unused. If AD RMS also unused, stop WID to free ~300 MB headroom. Mike awaiting Howard's verification before authorizing stop. 5. **[LOWER] Server 2019 migration scoping** — Three paths (component store repair + in-place; in-place without repair; clean build). Clean build is Mike's recommendation. Scope cost/timeline at next ACG strategy call before 2027-01-12 EOL. 6. **[LOWER] Documentation cleanup:** - Update workstation table in `docs/overview.md` with `DESKTOP-KRHQ5TS` / Manda / AIM USER#=4 - Confirm Manda's full name in AD - Disable SMB1 on IMC1 (`Set-SmbServerConfiguration -EnableSMB1Protocol $false`) - Drop `TestConv61223` DB on AIMSQL (leftover 2023 migration test) — safe per enumeration, but back up `.mdf` first - Clean up stale AD computer objects `IMC2`, `IMC-VM` --- ## History Highlights | Date | By | Event | |---|---|---| | ~2026-Q1 | Mike/Howard | Early engagement: 3 new workstations provisioned at Speedway (hostnames, AIM USER#s TBD in billing log) | | 2026-04-11/12/13 | Mike | IMC1 maintenance: RDS removal blocked (component store corruption 0x80073701), SSH installed, 716 GB freed on E: (backup cleanup), GFS retention automated, AIMsi DBs moved C:→S: SSD | | 2026-04-22 | Howard | Attempted remote domain-join of `DESKTOP-KRHQ5TS` over VPN — abandoned after subnet overlap + phantom DC defeated all workarounds | | 2026-04-28 | Howard | Onsite: `DESKTOP-KRHQ5TS` joined to imc.local, Manda (new GM) AD account created, Outlook/M365 configured, Office activated, AIMsi USER#=4 set. Ticket #32218, 1.5 hrs, prepay 14.0→12.5 hrs. ServerIMC confirmed as active authentication degrader. | | 2026-05-04 | Howard | Onsite (0.5 hrs): Station 2 receipt printer reconnected (re-added from \\imc1); VPN installed on Manda's machine. Ticket #32247. | | 2026-05-05 | Howard | AIM "connection broken" investigation. GuruRMM IMC client/site provisioned, IMC1 enrolled. Diagnosed memory pressure; scheduled AIMSQL restart for 02:30 (wrong instance — superseded next day). | | 2026-05-06 | Howard | Station 1 recurrence 12:14 PM. Full instance enumeration revealed SQLEXPRESS = production Standard (not AIMSQL). Wrong-instance restart task unregistered. Corrected diagnosis in session logs and PROJECT_STATE. Feedback memory created. | | 2026-05-07 | Mike | Decision: approved memory caps (SQLEXPRESS 12 GB, WID 512 MB, AIMSQL 256 MB), AIMSQL consolidation pending backup, Server 2016 migration timeline acknowledged, WSUS confirmed unused. | | 2026-07-07 | Howard | **Two-location split**: renamed GuruRMM site `IMCMain` -> **Main**, created **MOO** site (`SILVER-BRIDGE-5118`, key vaulted). Moved DESKTOP-GHG12G3 + IMC-LUIS to MOO; enrolled BigMom (was ScreenConnect-only) into MOO. Marked all machines in ScreenConnect (Company/Site + Front/Office location tags). Main = 14 agents, MOO = 3. | | 2026-07-07 (eve) | Howard | **Applied SQL memory caps** on IMC1 (were never applied since Mike's 2026-05-07 approval): SQLEXPRESS 12 GB, AIMSQL 256 MB, WID 512 MB — verified. WATCH probable-cause flag through ~2026-07-11. SQLEXPRESS/AIMSQL done as IMC\\guru (SYSTEM not sysadmin there). | | 2026-07-07 (eve) | Howard | **Fleet reconciliation + EDR rollout.** Confirmed AV = Datto EDR (not BD); deployed EDR to BigMom + DESKTOP-NFU17AJ (Datto org 16->18), queued EDR to IMC-LUIS/IMC-M-EdServices1/REPAIRADMIN. Enrolled DESKTOP-NFU17AJ in RMM (17->18), queued RMM install for Manda/Sara/PNVA9G51/LESLIE. Started REPAIRADMIN BD->EDR swap (EDR queued; BD removal pending EDR verify). Found MSP360 backup live-but-unbilled on IMC1; AV billed 20 seats. Identified 12 Syncro assets to archive (>40d offline). | --- ## Compilation Notes Source material: 5 session logs (2026-04-12 through 2026-05-06) + 1 decision file (2026-05-07) + README + PROJECT_STATE + 10 docs files (most docs/* are blank templates with no client-specific data filled in — network/firewall/vlans/VLAN/DHCP/DNS/RMM/AV/backup/issues docs are all empty templates). Many structured docs (`docs/network/`, `docs/security/`, `docs/cloud/`) are empty templates. The authoritative information sources are `README.md`, `PROJECT_STATE.md`, and the session logs. **Unverified items flagged:** - Whether Howard applied `max server memory` caps after Mike's 2026-05-07 approval - ServerIMC ticket status — ticket was recommended but not confirmed opened - Manda's full name in AD - M365 tenant details (tenant domain, license type, MFA policy) - WID instance AD RMS usage - AIMSQL `.mdf` file locations - Full workstation fleet AIM USER# assignments - ISP, firewall hardware, VLAN/network topology ## Backlinks - [[projects/gururmm]] — IMC1 enrolled as agent `fa99e913-1027-4e33-a928-7695e31068e7` - [[projects/gps-rmm-audit]] — full-fleet GuruRMM enrollment + the 2026-07-07 Main/MOO two-site split