diff --git a/clients/instrumental-music-center/session-logs/2026-07/2026-07-08-howard-imc-edr-rollout-sql-caps.md b/clients/instrumental-music-center/session-logs/2026-07/2026-07-08-howard-imc-edr-rollout-sql-caps.md new file mode 100644 index 00000000..0b0009c3 --- /dev/null +++ b/clients/instrumental-music-center/session-logs/2026-07/2026-07-08-howard-imc-edr-rollout-sql-caps.md @@ -0,0 +1,162 @@ +# IMC — fleet reconciliation, Datto EDR rollout, Bitdefender removal, SQL memory caps applied + +## User +- **User:** Howard Enos (howard) +- **Machine:** Howard-Home +- **Role:** tech + +## Session Summary + +Reconciled the Instrumental Music Center (IMC) fleet across GuruRMM, ScreenConnect, Datto +EDR, MSP360, and Syncro, then acted on the gaps. Started from a request to find IMC machines +in GuruRMM and retire (in Syncro) any machine not in RMM. Direction evolved: deploy GuruRMM to +the IMC ScreenConnect session group (default site Main, MOO only when clearly MOO by public +IP), and retire only Syncro assets offline > 40 days. + +Enrolled `DESKTOP-NFU17AJ` into GuruRMM/Main via the server's site-preconfigured installer +one-liner pushed through ScreenConnect `send-command` (verified: RMM agents 17 -> 18). Queued +the same install to four offline SC machines (Manda's `DESKTOP-KRHQ5TS`, Sara Sherer's +`SNAPPERLANDHOME`, `LAPTOP-PNVA9G51`, `LESLIE2019LAPTO`) — they run on reconnect. Determined +Syncro "retire" = "Archive" which is UI-only (no REST API); handed the user the 12-asset +archive list rather than deleting. + +Established that IMC's AV is Datto AV/EDR (16 agents), not Bitdefender (1 stray, `REPAIRADMIN`). +Billing check: recurring schedule #365662 ($1,590.02/mo) bills 20 AV seats but no backup line, +while MSP360 shows IMC1 actively backing up (Files + SQL daily; Images plan stale since +2024-10) — a backup revenue gap (user handling backups in a separate plan). Deployed Datto EDR +to the unprotected live machines: `BigMom` + `DESKTOP-NFU17AJ` via /rmm (installed, verified, +Datto org 16 -> 18), queued `IMC-LUIS`, `IMC-M-EdServices1`, and `REPAIRADMIN` via SC. Began the +`REPAIRADMIN` Bitdefender -> EDR swap (EDR queued; BD removal pending EDR verify, since BD has no +uninstall API and REPAIRADMIN is offline). + +Verified the outstanding "max server memory" caps (Mike-approved 2026-05-07) were never applied +— all three IMC1 instances were at the uncapped default (2147483647 MB). Applied them: WID 512 +MB as SYSTEM; SQLEXPRESS 12288 MB + AIMSQL 256 MB required `IMC\guru` (SQL sysadmin) via in- +process `LogonUser` + `RunImpersonated`, because the RMM agent's `NT AUTHORITY\SYSTEM` is not +sysadmin on those two instances. All three verified. Fixed `ps-encoded.sh` (added a Python +fallback for the UTF-16LE encode since this box lacks `iconv.exe`). Updated the IMC wiki +throughout and committed/pushed each change. + +## Key Decisions + +- **Site placement by public IP**, not guesswork: Main = 72.194.57.2, MOO = 68.107.177.170, + everything else -> Main (per user rule). MOO machines were already enrolled, so all new + enrollments went to Main. +- **Retire = Archive = UI-only**: confirmed against Syncro docs; refused to use + `DELETE /customer_assets` (destructive, loses history/ticket links). Handed the user the list. +- **Retirement threshold = offline > 40 days** (user), measured by ScreenConnect + `GuestInfoUpdateTime` because Syncro `updated_at` is a bulk-touch (unreliable). +- **AV-swap ordering**: install + verify Datto EDR BEFORE removing Bitdefender — never leave an + unprotected window. +- **SQL caps via IMC\guru impersonation over RMM** (user-approved credential path): fastest, but + the DA password transits one RMM command (base64) — flagged for optional rotation. +- **In-process token impersonation** for the SQL apply (not sqlcmd/Start-Process) — captures + output cleanly, no external tool dependency. +- **`ps-encoded.sh` Python fallback** rather than sourcing an `iconv.exe` binary — portable, + fixes the whole fleet, uses the already-present Python. + +## Problems Encountered + +- **SC `--company` filter returned empty** — CP1 is the full tagged value `IMC - Instrumental + Music Center`, not `Instrumental Music Center`; and CP values live under + `.CustomProperties.CustomPropertyN`. Resolved by enumerating via raw `CustomProperty1 = '...'`. +- **`jq` parse errors** on `rmm-search.sh --json` — caused by my `2>&1` merging the stderr + `[OK] Authenticated` banner into stdout. Fix: `2>/dev/null` when piping `--json` to jq. +- **`ps-encoded.sh` failed: `iconv: command not found`** — Git-for-Windows ships the iconv DLL + (`msys-iconv-2.dll`) but not `iconv.exe`, and has no pacman. Patched the script with a Python + fallback; verified byte-exact round-trip. +- **SQL RECONFIGURE permission denied on SQLEXPRESS + AIMSQL** — RMM agent runs as + `NT AUTHORITY\SYSTEM`, which is not sysadmin there (reduced to `IMC\guru` + AIM service acct on + 2026-04-12). Resolved by impersonating `IMC\guru` in-process. +- **Bitdefender has no uninstall API** (GravityZone console/local-tool only), and REPAIRADMIN has + anti-tampering ON — so removal will be BD's local uninstall tool via /rmm (blank uninstall + password per policy), pending the box coming online. + +## Configuration Changes + +- **IMC1 SQL** `max server memory (MB)`: SQLEXPRESS 2147483647 -> **12288**; AIMSQL 2147483647 -> + **256**; MSSQL$MICROSOFT##WID 2147483647 -> **512**. Reversible; no service restart. +- **GuruRMM**: `DESKTOP-NFU17AJ` enrolled (Instrumental Music Center / Main). 4 more queued via SC. +- **Datto EDR**: `BigMom` + `DESKTOP-NFU17AJ` agents installed (org Instrumental Music Center). + `IMC-LUIS`, `IMC-M-EdServices1`, `REPAIRADMIN` queued via SC. +- **`.claude/scripts/ps-encoded.sh`** — added Python fallback in `encode_b64()` (commit pushed). +- **`.claude/commands/syncro.md`** — added "Customer Assets (retire/archive = UI-only)" section. +- **`.claude/skills/screenconnect/SKILL.md`** — added verified GuruRMM/EDR deploy-via-send-command + recipe + gotchas. +- **`.claude/skills/datto-edr/SKILL.md`** — added SC-queue deploy for offline endpoints + AV-swap + ordering note. +- **`wiki/clients/instrumental-music-center.md`** — new Security/AV, Billing, fleet-recon sections; + SQL caps marked APPLIED with a WATCH/probable-cause flag through ~2026-07-11; Backups + Active + Work + History updated. +- **`errorlog.md`** — logged the retire=archive-not-delete correction and the iconv friction. + +## Credentials & Secrets + +- **`IMC\guru`** (Domain Admin + SQL sysadmin on SQLEXPRESS/AIMSQL): password in vault + `clients/imc/imc1.sops.yaml` field `credentials.domain.password`. Used this session via token + impersonation to apply the SQL caps. **It transited one GuruRMM command** (base64 in + `command_text`, cmd `378fffe7-a46b-4617-ae6c-38c8d34229b1`) — recoverable in the RMM DB; rotate + if zero residual exposure is required. Fallback password field also present: + `credentials.password-fallback`. +- **Datto EDR enrollment RegKey `instrume01`** — IMC scan-target group "IMC - Speedway (Loc1)". + Not currently vaulted (retrievable anytime via `edr.py deploy-cmd --group `); vault it if we + want it captured. +- No new credentials created this session. + +## Infrastructure & Servers + +- **IMC1** — 192.168.0.2, Windows Server 2016, 32 GB RAM (32560 MB reported). GuruRMM agent id + `88cbf7c0-abfa-4f12-846c-96274f718bff`. SQL instances: `SQLEXPRESS` (PROD, Standard, TCP 61151), + `AIMSQL` (orphan, Express, 63116), `MICROSOFT##WID`. +- **Public IPs:** Main/Speedway = 72.194.57.2; MOO = 68.107.177.170. +- **GuruRMM:** server `http://172.16.3.30:3001`; install page base `https://rmm.azcomputerguru.com`; + Main site code `INNER-BRIDGE-8354` (site id `2c5b65ad-2d5e-47b3-b12b-632e35e08ff6`); MOO + `SILVER-BRIDGE-5118` (`f9e4a897-708b-43c5-ab7d-a8b05decdc3e`). Client `Instrumental Music Center` + `213b62a8-30f4-41dd-9bb3-549341104416`. +- **Datto EDR:** tenant `azcomp4587.infocyte.com`; org `Instrumental Music Center` + `00b6b6e9-35ea-47cd-9413-088776434514`; scan-target group `IMC - Speedway (Loc1)` + `c56026f0-b174-4f84-9dc4-cf8cffa49b74`; RegKey `instrume01`. +- **Bitdefender:** IMC company `5e77afc47357486d522f1a44` (only endpoint `REPAIRADMIN` + `6a4d1bbe0a5a3122524a67ec`, policy `GPS Default` `5c42940b6e16d61a0c8b4568`, blank uninstall pw). +- **Syncro:** customer 7088508; recurring schedule 365662 ($1,590.02/mo). +- **MSP360:** company "Instrumental Music"; IMC1 plans — Files (daily, OK), SQL (daily, warnings), + Images (stale 2024-10-24). + +## Commands & Outputs + +- SQL verify (before): all three instances `max_server_memory_MB=2147483647` (uncapped default). +- SQL apply (after): `SQLEXPRESS: 12288 (verify=12288) as IMC\Guru`, `AIMSQL: 256 (verify=256) as + IMC\Guru`, `WID: 512 (verify=512)`. +- First SQL apply attempt (as SYSTEM) failed on SQLEXPRESS/AIMSQL: "User does not have permission + to run the RECONFIGURE statement." -> switched to IMC\guru impersonation. +- Datto EDR install (BigMom, NFU17AJ): exit 0, "Installed RTS agent to + C:\Program Files\infocyte\agent\agent.exe"; Datto org agents 16 -> 18. +- `ps-encoded.sh` fixed: encode/decode round-trip byte-exact (Python fallback). + +## Pending / Incomplete Tasks + +- **REPAIRADMIN Bitdefender -> Datto EDR swap** — EDR queued via SC; when online: verify EDR live, + push BD local uninstall tool via /rmm (confirm `uninstallTool.exe` path on the box), delete BD + console entry `6a4d1bbe0a5a3122524a67ec`. +- **EDR on the 4 SC-queued RMM enrollments** (`DESKTOP-KRHQ5TS`, `SNAPPERLANDHOME`, `LAPTOP-PNVA9G51`, + `LESLIE2019LAPTO`) — after they land in GuruRMM, push Datto EDR to each. +- **Verify the 4 queued GuruRMM installs + 2 queued EDR installs** (`IMC-LUIS`, `IMC-M-EdServices1`) + complete on reconnect. +- **User: archive the 12 offline>40d Syncro assets** (UI, Bulk Actions -> Archive), then clear the + 3 matching dead Datto agents (`desktop-urv3ugr`, `office1`, `purchasingcomp`). +- **WATCH through ~2026-07-11:** if IMC reports AIM/SQL/perf issues, treat the memory caps as the + first probable cause (rollback = uncap to 2147483647). +- **Backups** — live-but-unbilled on IMC1; SQL diff-chain churn vs the local 22:00 job; Images plan + stale. User handling in a separate plan. +- Optional: rotate `IMC\guru` (password transited RMM cmd 378fffe7); vault the Datto RegKey. + +## Reference Information + +- Wiki: `wiki/clients/instrumental-music-center.md`. +- GuruRMM install one-liner (Main): `iex (irm 'https://rmm.azcomputerguru.com/install/INNER-BRIDGE-8354/windows')`. +- Datto EDR install: `Install-EDR -URL "https://azcomp4587.infocyte.com" -RegKey instrume01`. +- RMM cmd ids: SQL apply (SYSTEM, partial) — check-memory `d2d918c2`; SQL apply (IMC\guru) + `378fffe7-a46b-4617-ae6c-38c8d34229b1`; EDR BigMom `f6653ac5`, NFU17AJ `07873cbe`. +- Syncro archive list (asset ids): 5345521, 5033192, 10333758, 2486998, 2967905, 3100580, 2451117, + 3151232, 3151234, 3151242, 3151229, 3150899. +- Commits pushed: wiki fleet-recon `840b0ce`; ps-encoded fix; wiki SQL-caps update (this session).