azcomputerguru/claudetools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
840b0ce9d650ca1b941dcf94a183020e931e27d4
claudetools/wiki/clients/instrumental-music-center.md
Howard Enos 840b0ce9d6 wiki(IMC): 2026-07-07 fleet reconciliation — Datto EDR rollout, AV vendor, backup/billing gap, retire list 
- AV/EDR is Datto (not Bitdefender); REPAIRADMIN BD->EDR swap in progress
- EDR deployed to BigMom + DESKTOP-NFU17AJ; queued LUIS/EdServices1/REPAIRADMIN
- DESKTOP-NFU17AJ enrolled in RMM (17->18); 4 more queued via SC
- MSP360 backup live-but-unbilled on IMC1; AV billed 20 seats
- 12 Syncro assets to archive (>40d offline; archive=UI-only)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 20:45:53 -07:00

420 lines
29 KiB
Markdown
Raw Blame History

---
type: client
name: instrumental-music-center
display_name: Instrumental Music Center
last_compiled: 2026-07-07
compiled_by: HOWARD-HOME/claude-main
sources:
- projects/gps-rmm-audit/session-logs/2026-07/2026-07-07-howard-gps-master-status-wikis-onsite-kit.md
- clients/instrumental-music-center/README.md
- clients/instrumental-music-center/PROJECT_STATE.md
- clients/instrumental-music-center/docs/overview.md
- clients/instrumental-music-center/docs/billing-log.md
- clients/instrumental-music-center/docs/2026-04-13-ticket-notes.md
- clients/instrumental-music-center/docs/network/topology.md
- clients/instrumental-music-center/docs/network/vlans.md
- clients/instrumental-music-center/docs/network/firewall.md
- clients/instrumental-music-center/docs/network/dhcp.md
- clients/instrumental-music-center/docs/network/dns.md
- clients/instrumental-music-center/docs/cloud/m365.md
- clients/instrumental-music-center/docs/cloud/azure.md
- clients/instrumental-music-center/docs/rmm/rmm.md
- clients/instrumental-music-center/docs/security/antivirus.md
- clients/instrumental-music-center/docs/security/backup.md
- clients/instrumental-music-center/docs/issues/log.md
- clients/instrumental-music-center/docs/servers/server_template.md
- clients/instrumental-music-center/session-logs/2026-04-12-imc1-cleanup-and-sql-move.md
- clients/instrumental-music-center/session-logs/2026-04-28-howard-manda-laptop-provision.md
- clients/instrumental-music-center/session-logs/2026-05-04-station2-printer-and-manda-vpn.md
- clients/instrumental-music-center/session-logs/2026-05-05-howard-aim-connection-broken-investigation.md
- clients/instrumental-music-center/session-logs/2026-05-06-howard-imc1-aim-instance-correction.md
- clients/instrumental-music-center/decisions/2026-05-07-mike-memory-allocation-approval.md
backlinks:
- projects/gururmm
---
# Instrumental Music Center
Music retail and instrument repair shop running AIMsi point-of-sale software on-prem. **Two locations:** the main store on Speedway (7063 E Speedway Blvd, Tucson AZ 85710) and a second location, **MOO** (6300 E El Dorado Cir, Tucson AZ 85715). ACG provides managed break-fix / prepaid-block support; primary focus is on the AIMsi SQL server (IMC1) and workstation fleet. In GuruRMM the two locations are the **Main** and **MOO** sites (see GuruRMM Enrollment).
---
## Profile
- **Contract type:** Prepaid hour block
- **Billing rate:** $175/hr all labor
- **Hours remaining:** 12.5 hrs as of 2026-04-28 (after debiting 1.5 hrs for Syncro #32218). Always live-check before billing.
- **Syncro customer ID:** 7088508
- **Key contacts:**
- **Leslie Stirm** — primary contact; leslie@imc-az.com; Syncro contact_id 731730
- **Manda** — General Manager (new, replaced Michael Santander as of ~2026-04-28). Full name unconfirmed in AD. [unverified]
- **Michael Santander** — former GM; domain account already deactivated.
- **Primary domain:** imc.local (on-prem AD)
- **Locations (2):** **Main** — Speedway (7063 E Speedway Blvd, Tucson AZ 85710); **MOO** — 6300 E El Dorado Cir, Tucson AZ 85715. Mapped as separate GuruRMM sites 2026-07-07.
- **Critical software:** AIMsi by Tri-Tech (https://www.tritechretail.com/topic/aim) — retail POS and inventory management.
---
## Infrastructure
### Servers & Services
| Host | IP | Role | OS | Notes |
|---|---|---|---|---|
| IMC1 | 192.168.0.2 | DC (imc.local), DNS, File Server, AIMsi SQL host, RDS host | Windows Server 2016 Standard (build 14393.7426) | Dell R720, 4 physical cores, 32 GB RAM. GuruRMM agent: `fa99e913-1027-4e33-a928-7695e31068e7` |
| ServerIMC | 192.168.0.63 | Phantom / broken DC | Windows Server 2016 Essentials [unverified] | **[WARNING] Registered as DC in AD DNS (A + SRV records for `_ldap._tcp.dc._msdcs.imc.local` and `_kerberos._tcp.imc.local`). Responds to ICMP but TCP/389 (LDAP) and TCP/88 (Kerberos) refuse connections. DC locator round-robins — clients that pick ServerIMC time out. Root cause of intermittent slow logons, GPO failures, and 2026-04-22 remote domain-join failure. Needs `ntdsutil` metadata cleanup (if demoted ghost) or AD service repair.** |
| IMC2 | — | Unknown (stale) | Windows Server 2016 Essentials | Last logon 2023 — likely decommissioned. Clean up AD computer object. |
| IMC-VM | — | Unknown (dead) | Windows Server 2016 Standard | Last logon 2021 — dead. Clean up AD computer object. |
| Station 1 | 192.168.0.50 | POS workstation | Windows [unverified] | Hostname `IMC-STATION1`. Primary workstation for AIM "connection broken" incidents. |
#### IMC1 SQL Instances (CRITICAL — read carefully)
**[WARNING] The production AIM database is on `IMC1\SQLEXPRESS`, NOT `IMC1\AIMSQL`. The instance name is actively misleading — someone installed SQL Server 2019 Standard under the default `SQLEXPRESS` instance name and never renamed it. This burned a full day of triage. Always verify SQL roles by active connections (`sys.dm_exec_sessions`) — never by instance name.**
| Instance | Port | Edition (actual) | Role | Production DB | Notes |
|---|---|---|---|---|---|
| `IMC1\SQLEXPRESS` | TCP 61151 | **SQL Server 2019 Standard** (misleading name!) | **PRODUCTION** | `IMCAIM` (created 2023-08-21) | Service account `IMC\AIM`. ~9 store workstations + 22 server-local AIM sessions. **Do not stop, do not uninstall.** ERRORLOG at `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Log\`. No `max server memory` cap (default unlimited). |
| `IMC1\AIMSQL` | TCP 63116 (dynamic) | SQL Server 2019 Express GDR 15.0.2165.1 | **Orphan** (consolidation candidate) | None active | Service account `IMC\IMC1$`. Zero established TCP connections. Holds only 2023-era conversion-test DBs (`AIM`, `IMC`, `TestConv61223`). No active backup chain landing here. Shutdown + uninstall approved by Mike pending `.mdf` backup confirmation. |
| `IMC1\MICROSOFT##WID` | — | Windows Internal Database | WSUS / AD RMS | — | WSUS confirmed NOT in use at IMC. AD RMS status unverified. If AD RMS also unused, instance can be stopped to free ~300 MB. **Canary for memory pressure** — Event 17890 paging events fire here first when the host is memory-squeezed. |
**Workstations connected to production `IMC1\SQLEXPRESS` (verified 2026-05-06):**
| Hostname | IP |
|---|---|
| IMC-MINI | 192.168.0.72 |
| IMC-SVCSTR | 192.168.0.55 |
| IMC-LESSONS | 192.168.0.62 |
| IMC-STATION2 | 192.168.0.66 |
| IMC-L1-STATION9 | 192.168.0.41 |
| DESKTOP-44L80C0 | 192.168.0.46 |
| DESKTOP-MR3ALTK | 192.168.0.59 |
| REPAIRADMIN | 192.168.0.48 |
| C2B | 192.168.0.4 |
| IMC-STATION1 | 192.168.0.50 |
All sessions authenticate as `AIMUser1` via `.Net SqlClient Data Provider`.
#### IMC1 Disk Layout
| Drive | Purpose | Notes |
|---|---|---|
| C: | OS, IIS, system DBs | 419 GB volume; ~278 GB used after 2026-04-12 cleanup (~66%); was 77% full before. Monitor. |
| E: | SQL backups + installers + Server 2016 media | `E:\W2016\sources\install.wim` is RTM 14393.0. SQL backups at `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\` |
| F: | Windows Image Backups | — |
| S: | Dedicated SSD (Samsung 850 PRO 256 GB) — AIMsi SQL user DBs | User DBs at `S:\SQL\Data\`. AIM client share `\\IMC1\AIM` → `S:\AIM`. System DBs remain on C:. |
### Email & Identity
- **Mail:** IMC uses a **mixed Google / Microsoft identity model** — different users are on different platforms. Manda is on the M365 side. [full tenant details unverified]
- **M365 tenant details:** Not fully documented. Manda's Outlook was configured against an existing M365 mailbox.
- **On-prem AD domain:** `imc.local`
- **MFA status:** [unverified]
- **DNS:** IMC1 (192.168.0.2) is the authoritative DNS server for imc.local. ServerIMC (192.168.0.63) has ghost A + SRV records as a DC — these are the direct cause of client authentication failures and need cleanup.
### Network
- **LAN subnet:** 192.168.0.0/24
- **VPN:** OpenVPN (.ovpn profile). **[WARNING] 192.168.0.0/24 subnet overlap hazard:** if technician's home/office LAN is also 192.168.0.0/24 (Howard's home is), OpenVPN routes win for reaching IMC1 but Windows multi-homed DNS races between the two interfaces. DNS negative caching causes domain join / locator failures. **If remote LAN overlaps IMC's subnet, go onsite for domain joins.** Also: disconnect Tailscale before connecting to IMC OpenVPN — Tailscale's `pfsense-2` subnet router advertises 192.168.0.0/24 with lower metric than the VPN, making IMC1 unreachable.
- **Firewall:** [unverified — not documented]
- **ISP:** [unverified]
- **SMB:** SMB1 still enabled on IMC1 — disable as security hygiene when opportunity permits.
- **SMB signing:** `RequireSecuritySignature = True` on server — adds auth overhead.
---
## GuruRMM Enrollment
GuruRMM client **Instrumental Music Center** (`213b62a8-30f4-41dd-9bb3-549341104416`, code
`IMC`). **Two sites, one per physical location** (created/split 2026-07-07):
| Site | Site ID | Site code | Enrollment key (vault) |
|---|---|---|---|
| **Main** (Speedway; renamed from `IMCMain`) | `2c5b65ad-2d5e-47b3-b12b-632e35e08ff6` | `INNER-BRIDGE-8354` | `clients/imc/gururmm-site-main.sops.yaml` |
| **MOO** (second location) | `f9e4a897-708b-43c5-ab7d-a8b05decdc3e` | `SILVER-BRIDGE-5118` | `clients/imc/gururmm-site-moo.sops.yaml` |
**Agent placement (as of 2026-07-07):**
- **MOO (3):** `BigMom`, `DESKTOP-GHG12G3`, `IMC-LUIS`
- **Main (14):** `IMC1`, `C2B`, `IMC-STATION1`, `IMC-STATION2`, `IMC-L1-STATION9`,
`DESKTOP-44L80C0` (a.k.a. "Office 2"), `DESKTOP-MR3ALTK` (a.k.a. "Office 1"), `IMC-Mini`,
`IMC-Lessons`, `IMC-SvcStr`, `IMC-M-EdServices1`, `RepairAdmin`, `LAPTOP-DCHQ3F92`,
`LAPTOP-UBTI0IE3`
IMC was first enrolled in GuruRMM 2026-05-05 (IMC1 only, via ScreenConnect). The rest of the
fleet was enrolled during the GPS -> GuruRMM audit (bulk SC push). On 2026-07-07 the client was
split into the Main + MOO sites, agents re-homed accordingly, and BigMom (previously SC-only)
enrolled into MOO. ScreenConnect metadata matches: Company = Instrumental Music Center, Site =
Main/MOO, with location tags (Front for the front-counter stations; Office 1/Office 2 for the
office desktops), Department kept as "Sales".
**Note:** When SSH from Howard-Home is blocked by the 192.168.0.0/24 route collision, GuruRMM remote commands are the fallback for running diagnostics on IMC1.
### 2026-07-07 fleet reconciliation (RMM / AV / backup / Syncro assets)
Cross-referenced GuruRMM vs ScreenConnect vs Datto EDR vs MSP360 vs Syncro assets for IMC.
**Site rule:** Main = public IP `72.194.57.2` (Speedway); MOO = `68.107.177.170`; anything
else defaults to Main. ScreenConnect CP1 = `IMC - Instrumental Music Center`, CP2 = Site
(SC truncates Name to 15 chars, e.g. `IMC-M-EDSERVICE` == RMM `IMC-M-EdServices1`).
**GuruRMM enrollments added/queued 2026-07-07 (all Main unless noted):**
- **Enrolled + verified:** `DESKTOP-NFU17AJ` (agent count 17 -> 18).
- **Queued via ScreenConnect** (single self-contained `iex (irm .../install/INNER-BRIDGE-8354/windows)`
in the session's one command slot; runs on reconnect): `DESKTOP-KRHQ5TS` (Manda),
`SNAPPERLANDHOME` (Sara Sherer), `LAPTOP-PNVA9G51`, `LESLIE2019LAPTO` (Leslie's events laptop).
## Security — AntiVirus / EDR
**IMC's AV/EDR is Datto AV/EDR (Infocyte), NOT Bitdefender.** Datto org **Instrumental Music
Center** `00b6b6e9-35ea-47cd-9413-088776434514`, single scan-target group **IMC - Speedway
(Loc1)** `c56026f0-b174-4f84-9dc4-cf8cffa49b74`, enrollment **RegKey `instrume01`** (all IMC
endpoints — incl. MOO — land in this one org/group). Deploy one-liner: `Install-EDR -URL
"https://azcomp4587.infocyte.com" -RegKey instrume01` (push via `/rmm`, or queue via SC
send-command for an offline box).
- **Bitdefender is being retired from IMC.** Only one IMC endpoint was ever on Bitdefender
GravityZone — `REPAIRADMIN` (BD company `5e77afc47357486d522f1a44`, BEST agent, anti-tamper
ON). **BD has NO uninstall API** (GravityZone console/local-tool only); policy has a blank
uninstall password (`shouldAskForPassword: false`), so removal = push BD's local uninstall
tool via `/rmm`. **Swap order: install + verify Datto EDR FIRST, then pull Bitdefender**
(no unprotected window).
- **EDR deployed 2026-07-07:** `BigMom` (MOO) + `DESKTOP-NFU17AJ` (Main) — installed, exit 0,
registered (Datto org 16 -> 18). **Queued via SC** (offline): `IMC-LUIS` (MOO),
`IMC-M-EdServices1` (Main), `REPAIRADMIN` (EDR then BD removal).
- **Still to cover:** the 4 SC-queued RMM enrollments above get EDR *after* they land in
GuruRMM (their one SC slot currently holds the RMM installer).
- **3 dead Datto agents** (`desktop-urv3ugr`, `office1`, `purchasingcomp`) = retire-list
machines; clear when their Syncro assets are archived.
## Billing (Syncro) — AV yes, Backup unbilled
Syncro customer **7088508**; one monthly recurring schedule **#365662** ("GPS/AV/365/Monthly
services - Leslie Stirm", $1,590.02/mo). Relevant lines:
- **AntiVirus: BILLED** — `GPS AntiVirus Add-on` 20 x $5 = **$100/mo** (20 seats). `GPS basic`
also bundles AV in its description — possible partial double-charge, worth a sanity check.
- **Backup: NOT billed** — no MSP360/Cloudberry/backup line on the invoice, and `GPS basic`'s
inclusion list does not cover backup. **IMC1 IS backing up (unbilled) — revenue gap.**
- Other lines: `GPS basic` 18 x $20, `GPS Pro Server` 2 x $80, Pax8 M365 (5), Hardware-Mgmt
fees, PacketDial VoIP.
- **AV seat reconciliation:** 20 billed vs ~14 actually protected (13 live Datto + 1 BD) at
audit time; the 2026-07-07 EDR rollout closes the gap by covering the previously-unprotected
live machines (`BigMom`, `IMC-LUIS`, `IMC-M-EdServices1`, `DESKTOP-NFU17AJ`, + REPAIRADMIN swap).
**Syncro asset retirement (offline > 40 days):** Syncro "retire" = **Archive** — **UI-only, no
REST API** (never `DELETE` — loses history/ticket links). 12 assets qualify (archive via Assets
tab -> Bulk Actions -> Archive): `PHIL2021LAPTOP` (5345521), `IMC-EVENTS` (5033192),
`DESKTOP-JQ0D38J` (10333758), `DESKTOP-URV3UGR` (2486998), `PURCHASINGCOMP` (2967905), `PHIL`
(3100580), `IMC-L1-GRAPHICS` (2451117), `DNW-L8-1..4` (3151232/234/242/229), `DNW-L24`
(3150899). Kept (< 40d): `LAPTOP-PNVA9G51`, `DESKTOP-KRHQ5TS`, `IMC-PRINTSERVER`.
---
## Access
- **SSH:** `ssh IMC\guru@192.168.0.2` — ed25519 key auth; PowerShell is the default shell. Authorized keys: `C:\ProgramData\ssh\administrators_authorized_keys` (inheritance off, Administrators + SYSTEM full control).
- **VPN:** OpenVPN (.ovpn profile). Disconnect Tailscale first. If home/office LAN is 192.168.0.0/24, remote domain operations will fail — go onsite instead.
- **Domain admin:** `IMC\guru` — also SQL sysadmin on both SQLEXPRESS and AIMSQL (added via single-user recovery 2026-04-12).
- **GuruRMM:** IMC1 agent `fa99e913-1027-4e33-a928-7695e31068e7` — use for remote commands when SSH is blocked.
- **Vault paths:**
- IMC1 credentials (domain admin, SSH): `clients/imc/imc1.sops.yaml`
- GuruRMM site enrollment key: `clients/imc/gururmm-site-main.sops.yaml`
**[WARNING] `sa` account on AIMSQL:** exists and enabled; password unknown. One candidate was tried and failed on 2026-04-12 — no lockout triggered (no lockout policy). If needed for AIMSQL consolidation, use single-user recovery mode (same process used 2026-04-12).
---
## AIMsi / Tri-Tech Critical Notes
**Per-machine workstation number (`USER#`) is mandatory.** AIMsi requires a user environment variable `USER#` (older Tri-Tech convention, still in use at IMC) set on each machine. This is the per-machine workstation identifier for POS polling and licensing.
- **NEVER wipe or reimage a machine without recording its `USER#` first.**
- **When deploying a new machine, assign its `USER#` per Leslie** — she tracks the allocation.
- Tri-Tech docs: https://www.tritechretail.com/topic/aim
**Known `USER#` assignments:**
| Machine | Hostname | USER# | Notes |
|---|---|---|---|
| Manda (GM) laptop | DESKTOP-KRHQ5TS | 4 | Assigned per Leslie, 2026-04-28 |
| Other workstations | Various | TBD | Not yet fully documented |
---
## Backups
- **Local SQL backups:** Nightly at 22:00 to `E:\SQL\MSSQL14.SQLEXPRESS\MSSQL\Backup\IMCAIM_*.bak`
- **Retention script:** `C:\Scripts\Clean-AimsiBackups.ps1` — GFS policy: 14 dailies + 1st-of-month; 3-newest safety override; logs to `C:\Scripts\Logs\aimsi-retention-YYYYMM.log`
- **Retention task:** `IMC AIMsi Backup Retention` — daily 23:30, SYSTEM, 1-hour limit
- **Off-site:** Cloudberry / MSP360 at `C:\ProgramData\Online Backup\`. Cloudberry chain confirmed intact before 2026-04-12 deletion run.
- SQLEXPRESS backup also confirmed landing at `C:\ProgramData\Online Backup\MSSQL\IMC1_SQLEXPRESS\`
- **MSP360 live status (verified 2026-07-07, company "Instrumental Music"):** IMC1 has 3 plans —
`Amisi Server - Files` **Success, ran daily** (last 2026-07-07 07:00, ~3.8 GB); `Backup MS SQL
Server` **ran daily but "completed with warnings"** (last 2026-07-07 21:00) — warning
`LsnDoNotMatchBeforeDiff` forces a full backup of `IMCAIM` because the local nightly SQL job
(22:00) and MSP360's diff chain compete for the log chain (align the two jobs to stop the churn);
`Amisi Server - Images` **stale — last ran 2024-10-24** (~20 mo; decide keep-or-drop). **Backup is
live but NOT billed** — see Billing. [Backup remediation tracked in a separate plan.]
- **Windows Image Backup:** on F:
- **AIMSQL orphan:** no backup chain. Locate and back up `AIM.mdf`, `IMC.mdf`, `TestConv61223.mdf` and their `.ldf` siblings before any consolidation — files were not found in expected path under `MSSQL15.AIMSQL\MSSQL\DATA` or `S:\*AIMSQL*` during 2026-05-06 search.
---
## Patterns & Known Issues
### [WARNING] Phantom DC `ServerIMC` — Active Authentication Degrader
`ServerIMC` (192.168.0.63) is registered in DNS as a domain controller (A record + SRV records for `_ldap._tcp.dc._msdcs.imc.local` and `_kerberos._tcp.imc.local`) alongside IMC1. It responds to ICMP ping but TCP/389 and TCP/88 refuse connections. The DC locator round-robins between IMC1 and ServerIMC, timing out ~50% of the time.
**Effect:** Intermittent slow logons, GPO failures, and broken remote domain joins for every domain client at IMC. Was the confirmed root cause of the 2026-04-22 failed remote join of `DESKTOP-KRHQ5TS`.
**Action needed:** Open a ticket. Either:
1. Repair AD services if `ServerIMC` is a real machine with broken services, or
2. Run `ntdsutil` metadata cleanup if it is a ghost from a previously demoted DC.
This was first flagged as "unclear" on 2026-04-13, promoted to confirmed issue 2026-04-28. No ticket has been opened as of 2026-05-06.
### AIM "Connection Broken" — Memory Pressure on IMC1
**Symptom:** `Telerik.OpenAccess.RT.sql.SQLException: Connection has been closed / The connection is broken and recovery is not possible` — user-facing AIM crash. First seen 2026-05-05 on Station 1 (IMC-STATION1, 192.168.0.50), recurred 2026-05-06 ~12:14 PM.
**Root cause:** IMC1 is hosting DC services + 6 concurrent RDP users + AIMsi Webservice/Runtime + three SQL instances + QuickBooks Enterprise on 32 GB. Under memory pressure, Windows trims SQL working sets (visible as WID Event 17890 paging events — the canary). The trim reaps idle Telerik OpenAccess TCP pool slots. Telerik has no transient-fault retry, so the next query against a dead pool handle throws the raw stack trace.
**SQLEXPRESS has no `max server memory` cap** (default 2,147,483,647 MB). Working set observed at 6.86 GB.
**Approved fix (Mike, 2026-05-07):** Cap `max server memory` on each instance:
- `SQLEXPRESS`: 12,288 MB (12 GB)
- `MSSQL$MICROSOFT##WID`: 512 MB
- `MSSQL$AIMSQL`: 256 MB (or consolidate it)
**Status as of 2026-05-06:** Howard is awaiting go-ahead for implementation. Mike approved on 2026-05-07. **Confirm whether Howard has applied the caps — this is the immediate recurrence prevention.** [unverified post-2026-05-07]
### [WARNING] SQL Instance Name Trap
**`IMC1\SQLEXPRESS` is SQL Server 2019 Standard Edition** — someone installed Standard under the default `SQLEXPRESS` instance name and never renamed it. `SERVERPROPERTY('Edition')` is the only way to confirm this. The instance name actively misleads.
**Never assume an instance is idle, orphan, or Express based on name.** Always verify by:
1. `SERVERPROPERTY('Edition')` for edition
2. `sys.dm_exec_sessions` for active user sessions
3. `Get-NetTCPConnection -OwningProcess` for established TCP connections
This trap caused a wrong-instance restart task to be deployed (2026-05-05) that had zero effect on the user-facing problem and was unregistered the next day (2026-05-06). See `.claude/memory/feedback_sql_instance_role_by_connection.md`.
### Component Store Corruption on IMC1 (RDS Removal Blocked)
`COMPONENTS` registry hive is ~168 MB (normal 30-50 MB), causing `0x80073701 ERROR_SXS_ASSEMBLY_MISSING` on any role removal or CU apply-on-boot. ETW manifest for provider GUID `{9c2a37f3-e5fd-5cae-bcd1-43dafeee1ff0}` is malformed — causes `CBS_E_INSTALLERS_FAILED` → full rollback even when CU staging succeeds.
**Effect:** Blocks RDS role removal, which was the original reason for the 2026-04-12 engagement. Also means CU KB5075999 cannot be applied cleanly.
**Server is otherwise healthy** — AIMsi production is running. This is a structural impediment to the Server 2019 migration. Three paths considered (see History Highlights).
### Remote Domain Join Over OpenVPN — Don't Do It
If the technician's local LAN subnet overlaps IMC's 192.168.0.0/24, remote domain joins over OpenVPN will fail reliably:
- OpenVPN pushed routes win for TCP, but Windows multi-homed DNS races between LAN DNS and VPN DNS (both respond to `imc.local` queries; LAN returns NXDOMAIN faster; Windows caches the negative answer).
- Even with NRPT rules, hosts file entries, `-Server <IP>` on Add-Computer, and `nltest /dsgetdc /force` — the combination of subnet overlap + phantom DC (ServerIMC) beat all client-side workarounds.
**Rule:** For IMC domain operations where local subnet overlap exists, go onsite.
### Mixed Email Identity (Google + M365)
IMC users are split between Google Workspace and Microsoft 365 — different users on different platforms. When configuring a new user, confirm with Leslie which platform their mailbox lives on before setting up Outlook vs. Gmail.
### Stale AD Objects
| Object | Last Logon | Status | Action |
|---|---|---|---|
| IMC2 (computer) | 2023 | Likely decommissioned | Clean up AD object |
| IMC-VM (computer) | 2021 | Dead | Clean up AD object |
| ServerIMC (DC) | Active (ICMP) | Phantom/broken DC | ntdsutil metadata cleanup or repair |
### GPO Noise
- **DistributedCOM 10016** fires every 5 minutes — RuntimeBroker permission noise. Cosmetic.
- **Group Policy event 103** fires every 5 minutes — "removal of the assignment of application Syncro from policy Management SW failed". Stale GPO object. Cleanup separately.
### Server 2016 EOL
Extended support ends **2027-01-12**. Migration window is finite. The memory pressure / AIM reliability incident is additional evidence to push the migration timeline. Mike wants to scope cost/timeline at next ACG strategy call.
---
## Active Work
As of 2026-07-07 (fleet reconciliation):
1. **[PENDING] REPAIRADMIN Bitdefender -> Datto EDR swap** — EDR install queued via SC; when
the box next comes online, verify EDR live, then push BD's local uninstall tool via `/rmm`
(blank uninstall password) and delete the stale BD console entry.
2. **[PENDING] EDR on the SC-queued RMM enrollments** — after `DESKTOP-KRHQ5TS` (Manda),
`SNAPPERLANDHOME` (Sara), `LAPTOP-PNVA9G51`, `LESLIE2019LAPTO` land in GuruRMM, push Datto
EDR to each (their one SC slot currently holds the RMM installer).
3. **[PENDING - user] Archive 12 offline-> 40d Syncro assets** — UI-only (Assets -> Bulk
Actions -> Archive); list in Billing section above. Then remove the 3 matching dead Datto
agents.
4. **[SEPARATE PLAN] Backup billing + SQL diff-chain churn** — IMC1 backs up but isn't billed;
`Amisi Server - Images` plan stale since 2024-10; SQL diff chain competes with the local
22:00 job. Being handled in a separate backup plan.
As of 2026-05-07 (last decision recorded):
1. **[IMMEDIATE] Apply `max server memory` caps on IMC1 SQL instances** — Mike approved 2026-05-07. Howard to implement: SQLEXPRESS 12 GB, WID 512 MB, AIMSQL 256 MB. Reversible (1-second config change, no service restart). Until applied, AIM connection-broken errors will continue recurring. [unverified — confirm applied]
2. **[HIGH] Open ticket for ServerIMC phantom DC investigation** — SRV/A records in DNS claim it's a DC; LDAP/Kerberos refuse connections. Degrades authentication for all domain users. No ticket opened as of 2026-05-06.
3. **[MEDIUM] AIMSQL orphan consolidation** — Mike approved (2026-05-07). Pending:
- Locate `AIM.mdf`, `IMC.mdf`, `TestConv61223.mdf` and `.ldf` siblings (not in expected path)
- Back up 2023-era DBs before shutdown
- Verify no applications reference `IMC1\AIMSQL` (TCP 63116)
- Stop and uninstall `MSSQL$AIMSQL`
4. **[MEDIUM] WID instance decision** — Verify AD RMS usage. WSUS confirmed unused. If AD RMS also unused, stop WID to free ~300 MB headroom. Mike awaiting Howard's verification before authorizing stop.
5. **[LOWER] Server 2019 migration scoping** — Three paths (component store repair + in-place; in-place without repair; clean build). Clean build is Mike's recommendation. Scope cost/timeline at next ACG strategy call before 2027-01-12 EOL.
6. **[LOWER] Documentation cleanup:**
- Update workstation table in `docs/overview.md` with `DESKTOP-KRHQ5TS` / Manda / AIM USER#=4
- Confirm Manda's full name in AD
- Disable SMB1 on IMC1 (`Set-SmbServerConfiguration -EnableSMB1Protocol $false`)
- Drop `TestConv61223` DB on AIMSQL (leftover 2023 migration test) — safe per enumeration, but back up `.mdf` first
- Clean up stale AD computer objects `IMC2`, `IMC-VM`
---
## History Highlights
| Date | By | Event |
|---|---|---|
| ~2026-Q1 | Mike/Howard | Early engagement: 3 new workstations provisioned at Speedway (hostnames, AIM USER#s TBD in billing log) |
| 2026-04-11/12/13 | Mike | IMC1 maintenance: RDS removal blocked (component store corruption 0x80073701), SSH installed, 716 GB freed on E: (backup cleanup), GFS retention automated, AIMsi DBs moved C:→S: SSD |
| 2026-04-22 | Howard | Attempted remote domain-join of `DESKTOP-KRHQ5TS` over VPN — abandoned after subnet overlap + phantom DC defeated all workarounds |
| 2026-04-28 | Howard | Onsite: `DESKTOP-KRHQ5TS` joined to imc.local, Manda (new GM) AD account created, Outlook/M365 configured, Office activated, AIMsi USER#=4 set. Ticket #32218, 1.5 hrs, prepay 14.0→12.5 hrs. ServerIMC confirmed as active authentication degrader. |
| 2026-05-04 | Howard | Onsite (0.5 hrs): Station 2 receipt printer reconnected (re-added from \\imc1); VPN installed on Manda's machine. Ticket #32247. |
| 2026-05-05 | Howard | AIM "connection broken" investigation. GuruRMM IMC client/site provisioned, IMC1 enrolled. Diagnosed memory pressure; scheduled AIMSQL restart for 02:30 (wrong instance — superseded next day). |
| 2026-05-06 | Howard | Station 1 recurrence 12:14 PM. Full instance enumeration revealed SQLEXPRESS = production Standard (not AIMSQL). Wrong-instance restart task unregistered. Corrected diagnosis in session logs and PROJECT_STATE. Feedback memory created. |
| 2026-05-07 | Mike | Decision: approved memory caps (SQLEXPRESS 12 GB, WID 512 MB, AIMSQL 256 MB), AIMSQL consolidation pending backup, Server 2016 migration timeline acknowledged, WSUS confirmed unused. |
| 2026-07-07 | Howard | **Two-location split**: renamed GuruRMM site `IMCMain` -> **Main**, created **MOO** site (`SILVER-BRIDGE-5118`, key vaulted). Moved DESKTOP-GHG12G3 + IMC-LUIS to MOO; enrolled BigMom (was ScreenConnect-only) into MOO. Marked all machines in ScreenConnect (Company/Site + Front/Office location tags). Main = 14 agents, MOO = 3. |
| 2026-07-07 (eve) | Howard | **Fleet reconciliation + EDR rollout.** Confirmed AV = Datto EDR (not BD); deployed EDR to BigMom + DESKTOP-NFU17AJ (Datto org 16->18), queued EDR to IMC-LUIS/IMC-M-EdServices1/REPAIRADMIN. Enrolled DESKTOP-NFU17AJ in RMM (17->18), queued RMM install for Manda/Sara/PNVA9G51/LESLIE. Started REPAIRADMIN BD->EDR swap (EDR queued; BD removal pending EDR verify). Found MSP360 backup live-but-unbilled on IMC1; AV billed 20 seats. Identified 12 Syncro assets to archive (>40d offline). |
---
## Compilation Notes
Source material: 5 session logs (2026-04-12 through 2026-05-06) + 1 decision file (2026-05-07) + README + PROJECT_STATE + 10 docs files (most docs/* are blank templates with no client-specific data filled in — network/firewall/vlans/VLAN/DHCP/DNS/RMM/AV/backup/issues docs are all empty templates).
Many structured docs (`docs/network/`, `docs/security/`, `docs/cloud/`) are empty templates. The authoritative information sources are `README.md`, `PROJECT_STATE.md`, and the session logs.
**Unverified items flagged:**
- Whether Howard applied `max server memory` caps after Mike's 2026-05-07 approval
- ServerIMC ticket status — ticket was recommended but not confirmed opened
- Manda's full name in AD
- M365 tenant details (tenant domain, license type, MFA policy)
- WID instance AD RMS usage
- AIMSQL `.mdf` file locations
- Full workstation fleet AIM USER# assignments
- ISP, firewall hardware, VLAN/network topology
## Backlinks
- [[projects/gururmm]] — IMC1 enrolled as agent `fa99e913-1027-4e33-a928-7695e31068e7`
- [[projects/gps-rmm-audit]] — full-fleet GuruRMM enrollment + the 2026-07-07 Main/MOO two-site split
Reference in New Issue View Git Blame Copy Permalink