From 5ec20ac9dd725c36cfed018912609e59d97c07bb Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Thu, 23 Apr 2026 09:19:23 -0700 Subject: [PATCH] session log: Dataforth SMTP fix, GuruRMM GAGETRAK onboarding, Cloudflare grey-cloud, ticket #32142 billed - Resolved calibration@dataforth.com SMTP AUTH per-mailbox block in Exchange Online - Full Dataforth tenant onboarding (all 5 ComputerGuru apps consented) - GuruRMM agent deployed on DF-GAGETRAK; diagnosed and fixed two issues: - rmm-api.azcomputerguru.com grey-clouded (Cloudflare was blocking WSS) - enrolled_agents auth gap workaround (site API key in AgentKey registry) - Syncro ticket #32142 billed: 2 hrs prepaid, invoice #67447, status Invoiced - syncro.md: fix .comment.id jq path (was .id, caused duplicate comments twice) - tenants.md: Dataforth marked fully onboarded Co-Authored-By: Claude Sonnet 4.6 --- .claude/commands/syncro.md | 18 +- .../remediation-tool/references/tenants.md | 2 +- .../session-logs/2026-04-23-session.md | 252 ++++++++++++++++++ 3 files changed, 270 insertions(+), 2 deletions(-) create mode 100644 clients/dataforth/session-logs/2026-04-23-session.md diff --git a/.claude/commands/syncro.md b/.claude/commands/syncro.md index 02f9935..8ec5133 100644 --- a/.claude/commands/syncro.md +++ b/.claude/commands/syncro.md @@ -140,7 +140,23 @@ echo "Authenticated as: $ME" **Silently ignored (do not use):** `product_id`, `minutes_spent`, `bill_time_now` — accepted but not saved. Verified 2026-04-21. -**CRITICAL — duplicate prevention:** The server has no idempotency. One POST = one comment, always. Duplicates are caused by calling the endpoint twice (retry after a perceived timeout, double tool invocation, etc.). **Never retry a POST /comment without first GET /tickets/{id} to confirm the comment did not already land.** The `Idempotency-Key` header is silently ignored. +**CRITICAL — response wrapper:** POST /comment returns `{"comment": {"id": ..., "subject": ..., ...}}` — NOT a flat object. Always parse as `.comment.id`, `.comment.created_at`, etc. Using `.id` returns null and looks like failure even when the comment posted successfully. This caused duplicate comments on 2026-04-22 (#32185) and 2026-04-23 (#32142) — both times the POST succeeded but null `.id` triggered a retry. + +```bash +# Correct pattern — always check .comment.id +RESP=$(curl -s -X POST "${BASE}/tickets/${ID}/comment?api_key=${API_KEY}" \ + -H "Content-Type: application/json" \ + -d @/tmp/payload.json) +echo "$RESP" | jq '{id: .comment.id, subject: .comment.subject, created_at: .comment.created_at}' +``` + +**CRITICAL — duplicate prevention:** The server has no idempotency. One POST = one comment, always. Duplicates are caused by calling the endpoint twice (retry after a perceived timeout, double tool invocation, etc.). **Never retry a POST /comment without first GET /tickets/{id} to confirm the comment did not already land.** When verifying, search all comments by subject — do not rely on `[-3:]` tail. The `Idempotency-Key` header is silently ignored. + +```bash +# Correct verification pattern after ambiguous response +curl -s "${BASE}/tickets/${ID}?api_key=${API_KEY}" | \ + jq '.ticket.comments[] | select(.subject == "Your Subject Here") | {id, created_at}' +``` **Comments cannot be deleted via API.** No DELETE endpoint exists in the Syncro API for comments — confirmed against official swagger spec. Duplicate comments require manual removal in the GUI. diff --git a/.claude/skills/remediation-tool/references/tenants.md b/.claude/skills/remediation-tool/references/tenants.md index f0f7049..fd3f1cf 100644 --- a/.claude/skills/remediation-tool/references/tenants.md +++ b/.claude/skills/remediation-tool/references/tenants.md @@ -18,7 +18,7 @@ After full onboarding, update the Onboarded column below. | CUADRO LLC | cuadro.design | b68c7171-31d6-4b63-8243-7a2cade9caf8 | NO | | | Curtis Plumbing | cparizona.onmicrosoft.com | d2d7ea54-9146-42d1-b99e-0da098550bde | NO | | | cwconcretellc.com | NETORGFT11452752.onmicrosoft.com | dfee2224-93cd-4291-9b09-6c6ce9bb8711 | NO | | -| Dataforth Corporation | dataforth.com | 7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584 | NO | Old app only | +| Dataforth Corporation | dataforth.com | 7dfa3ce8-c496-4b51-ab8d-bd3dcd78b584 | YES | All apps consented; Sec Inv + Exch Op + User Mgr roles assigned 2026-04-23; Exch Op Exchange Admin role added manually | | Feline Limited Cat Clinic | felineltd.onmicrosoft.com | 1b5f38ef-b6c8-4b6d-9bfb-9250ea7e7994 | NO | | | Glaz-Tech Industries | glaztech.com | 82931e3c-de7a-4f74-87f7-fe714be1f160 | NO | | | Grabblaw | grabblaw.com | 032b383e-96e4-491b-880d-3fd3295672c3 | YES | Sec Inv + User Mgr + Tenant Admin consented; all roles assigned 2026-04-20 | diff --git a/clients/dataforth/session-logs/2026-04-23-session.md b/clients/dataforth/session-logs/2026-04-23-session.md new file mode 100644 index 0000000..706fca5 --- /dev/null +++ b/clients/dataforth/session-logs/2026-04-23-session.md @@ -0,0 +1,252 @@ +# Session Log — Dataforth Corporation +**Date:** 2026-04-23 +**Duration:** Multi-hour session +**Type:** Client work — M365 investigation, SMTP fix, GuruRMM agent deployment + +## User +- **User:** Mike Swanson (mike) +- **Machine:** DESKTOP-0O8A1RL +- **Role:** admin + +--- + +## Session Summary + +Investigated and resolved the Gagetrak application email failure for Dataforth. Root cause was SMTP AUTH explicitly disabled at the per-mailbox level on calibration@dataforth.com in Exchange Online. Fixed via Exchange Online REST API. Also performed full M365 tenant onboarding for all ComputerGuru apps, deployed GuruRMM agent on DF-GAGETRAK, diagnosed and fixed two GuruRMM infrastructure issues, and billed/closed the Syncro ticket. + +--- + +## Work Completed + +### 1. M365 Tenant Investigation (calibration@dataforth.com) + +**Goal:** Diagnose why Gagetrak application couldn't send email via calibration@dataforth.com. + +**Tenant:** dataforth.com +**Tenant ID:** (resolved from domain via Graph API) + +**Findings:** +- Account status: Enabled, sign-in allowed +- MFA: Not enrolled on calibration@ (no app passwords needed) +- SMTP AUTH tenant-wide: null (not disabled at tenant level) +- SMTP AUTH per-mailbox: **SmtpClientAuthenticationDisabled = true** — this was the root cause +- License: M365 Business Standard (includes Exchange Online, desktop Office apps) +- AllowedToCreateAppPasswords: not required (no MFA) + +**Fix applied:** +``` +PATCH /adminapi/beta/{tenant}/CasMailbox/calibration@dataforth.com +{"SmtpClientAuthenticationDisabled": false} +``` +Verified: re-read showed `SmtpClientAuthenticationDisabled = false` after ~8 second replication delay. + +### 2. Full Dataforth Tenant Onboarding (All ComputerGuru Apps) + +Ran `onboard-tenant.sh` for dataforth.com. All 5 ComputerGuru apps consented: + +| App | Tier | Status | +|---|---|---| +| ComputerGuru Security Investigator | investigator | Consented | +| ComputerGuru Exchange Operator | exchange-op | Consented | +| ComputerGuru User Manager | user-manager | Consented | +| ComputerGuru Tenant Admin | tenant-admin | Consented | +| ComputerGuru Defender Add-on | defender | Consented | + +**Additional manual step required (gap in onboard-tenant.sh):** +- Exchange Operator SP (b43e7342) needed Exchange Administrator directory role assigned +- onboard-tenant.sh only assigns Exchange Admin to Security Investigator, not Exchange Operator +- Assigned manually via Graph API using Tenant Admin token +- Filed as future fix for the script + +**tenants.md updated:** Dataforth row changed from `NO | Old app only` to `YES | All apps consented; Sec Inv + Exch Op + User Mgr roles assigned 2026-04-23; Exch Op Exchange Admin role added manually` + +### 3. calibration@ Password Reset + +Reset via User Manager tier (Graph API): +- **New password:** `lMRCN#o2uP3$cwuoKIx0` +- Kevin (kwackerly) handles the account — notified via Syncro ticket comment + +### 4. GuruRMM Agent on DF-GAGETRAK + +**Machine:** DF-GAGETRAK, 192.168.0.102, Windows 11 Pro +**Access method:** WinRM double-hop (local → AD2 192.168.0.6 → GAGETRAK 192.168.0.102) + +**Install process:** +- Server at 172.16.3.30 not reachable from Dataforth network (internal IP) +- Downloaded MSI via local machine, copied bytes through WinRM sessions +- MSI exit code 1603 (already installed — GuruRMMAgent service pre-existed from Apr 21) +- Set registry keys via WinRM: + +``` +HKLM:\SOFTWARE\GuruRMM\ + SiteId = 3a2f6866-26cd-452c-9806-a8df21475c3c (Dataforth D1) + ApiKey = grmm_b-UKf4zCGx9jRiiCUCLtjuL6Fmemms9f + Version = 0.6.2 + InstallDir = C:\Program Files\GuruRMM\ +``` + +**Enrollment:** +- Called POST /api/enroll on server → HTTP 201 +- AgentKey returned: `agk_nW7msMhO6-LgURI7F3YYEWVnV6JMpjMW` (written to registry by agent) +- enrolled_agents record: `bf87301e-b4eb-4685-8479-a6215fc7195f` + +**Agent not coming online — diagnosis:** + +Issue 1 — Cloudflare blocking WebSocket: +- rmm-api.azcomputerguru.com was orange-clouded → Cloudflare accepted TCP but didn't forward WSS +- Fixed: set DNS to grey-cloud (DNS-only) via Cloudflare API +- Record: `30b183f66e1c24faf898f8b3efc9547d` → `proxied: false`, TTL 60 +- Server real IP: 72.194.62.10 + +Issue 2 — enrolled_agents keys not validated by WebSocket auth: +- Server ws/mod.rs authenticate() only checks `sites` table and `agents` table (legacy) +- It never checks `enrolled_agents.agent_key_hash` +- Agent sending `AgentKey = agk_...` → "Invalid API key" loop +- **Workaround:** Wrote site API key into AgentKey registry field: + ``` + AgentKey = grmm_b-UKf4zCGx9jRiiCUCLtjuL6Fmemms9f + ``` +- Agent now authenticates via site-based path, auto-registers against Dataforth D1 site + +**DF-GAGETRAK online:** +- agents table: `7626d82c-0736-47a6-8bc6-68e39859caed` +- hostname: DF-GAGETRAK +- device_id: `win-901ce38b-fb6e-44b8-a577-7c0bdf269a9a` +- site_id: `3a2f6866-26cd-452c-9806-a8df21475c3c` (Dataforth D1) +- status: online +- os_type: windows + +**Log file on GAGETRAK:** `C:\ProgramData\GuruRMM\agent.log.2026-04-23` + +### 5. GuruRMM Issues Filed (Gitea) + +- **Issue #8:** `bug: enrolled_agents keys never validated by WebSocket auth — agent loops Invalid API key` + - Fix: add third auth path in ws/mod.rs authenticate() that checks enrolled_agents.agent_key_hash +- **Issue #9:** `rmm-api.azcomputerguru.com must be grey-clouded — Cloudflare proxy blocks WebSocket` + - Options: keep grey-cloud, Cloudflare Tunnel, or fix nginx WebSocket proxy config + +### 6. Syncro Ticket #32142 — Billing and Close + +**Ticket:** #32142 (ID: 108919783) — "Remote - Error message from Gagetrak" +**Customer:** Dataforth Corporation (customer_id: 578095) + +**Resolution comment posted** (public, ID: 407135346): +- Root cause explanation (per-mailbox SMTP AUTH disabled) +- SMTP settings for Gagetrak app +- Password for calibration@ included + +**DUPLICATE COMMENT:** ID 407135146 (09:05) is a duplicate of 407135346 (09:06). +**ACTION REQUIRED:** Delete comment 407135146 manually in Syncro GUI. + +**Billing:** +- 2 hours Labor - Remote Business (product_id: 1190473) +- Line item ID: 42144896 +- Invoice #67447 (ID: 1650026361) +- **NOTE:** Invoice shows $0.00 — Dataforth is on prepaid hours. 2 hours consumed from their block. +- Ticket status: Invoiced + +### 7. Gagetrak SMTP Configuration (Pending Kevin) + +Kevin (kwackerly) needs to configure Gagetrak application with: +``` +Server: smtp.office365.com +Port: 587 +Security: STARTTLS / Explicit SSL +Username: calibration@dataforth.com +Password: lMRCN#o2uP3$cwuoKIx0 +``` + +### 8. Office 365 Pre-install Attempt (Blocked) + +Attempted to pre-install Office 365 on DF-GAGETRAK via WinRM. +**Blocked:** EDR on DESKTOP-0O8A1RL flagged the WinRM Invoke-Command with credentials as malicious lateral movement and blocked PowerShell execution. + +**Plan:** Kevin installs Office himself from office.com using his M365 Business Standard account. He signs in → Install apps → Microsoft 365 apps. No pre-staging needed since he must activate with his credentials anyway. + +ODT binary staged on server at `/var/www/gururmm/downloads/odt.exe` (v19929-20062, 3.4MB) if needed for future deployments. + +--- + +## Infrastructure Changes + +### Cloudflare DNS +- **rmm-api.azcomputerguru.com:** Changed from orange-cloud (proxied) to grey-cloud (DNS-only) +- Points to: 72.194.62.10 +- TTL: 60 seconds +- Cloudflare record ID: 30b183f66e1c24faf898f8b3efc9547d +- Zone ID: 1beb9917c22b54be32e5215df2c227ce +- **Do NOT re-enable orange cloud** without identifying root cause — will break all external agent connections + +### GuruRMM Database (PostgreSQL on 172.16.3.30) +- New agent record created for DF-GAGETRAK: `7626d82c-0736-47a6-8bc6-68e39859caed` +- enrolled_agents record remains: `bf87301e-b4eb-4685-8479-a6215fc7195f` (agent_id still NULL — not linked) + +--- + +## Credentials + +| Service | Detail | +|---|---| +| Dataforth calibration@ password | `lMRCN#o2uP3$cwuoKIx0` | +| Dataforth AD2 (INTRANET\sysadmin) | `Paper123!@#` | +| GuruRMM server SSH (guru@172.16.3.30) | `Gptf*77ttb123!@#-rmm` | +| GuruRMM server sudo password | same as SSH | +| GuruRMM PostgreSQL (gururmm) | `43617ebf7eb242e814ca9988cc4df5ad` | +| Gitea API token | `9b1da4b79a38ef782268341d25a4b6880572063f` | +| Cloudflare API token | `DRRGkHS33pxAUjQfRDzDeVPtt6wwUU6FwtXqOzNj` | +| Dataforth D1 site API key | `grmm_b-UKf4zCGx9jRiiCUCLtjuL6Fmemms9f` | +| Dataforth D1 site_id | `3a2f6866-26cd-452c-9806-a8df21475c3c` | + +--- + +## Key File Paths + +| Path | Location | Purpose | +|---|---|---| +| `D:\claudetools\.claude\commands\syncro.md` | local | Updated: .comment.id jq fix, verification pattern | +| `D:\claudetools\.claude\skills\remediation-tool\references\tenants.md` | local | Updated: Dataforth row | +| `/var/www/gururmm/downloads/odt.exe` | 172.16.3.30 | Office ODT v19929-20062 staged | +| `/home/guru/gururmm/server/src/ws/mod.rs` | 172.16.3.30 | authenticate() — enrolled_agents gap (see Issue #8) | +| `C:\ProgramData\GuruRMM\agent.log.2026-04-23` | DF-GAGETRAK | Agent connection logs | +| `C:\Program Files\GuruRMM\gururmm-agent.exe` | DF-GAGETRAK | Agent binary (0.6.2, Apr 21 2026) | + +--- + +## Pending Tasks + +- [ ] **[URGENT]** Delete duplicate Syncro comment ID 407135146 on ticket #32142 via GUI +- [ ] Kevin to configure Gagetrak SMTP and confirm email sends (ticket #32142 — Waiting on Customer after confirm) +- [ ] Kevin to install Office 365 from office.com on DF-GAGETRAK +- [ ] Fix onboard-tenant.sh gap: Exchange Admin role not assigned to Exchange Operator SP +- [ ] GuruRMM Issue #8: Add enrolled_agents WebSocket auth path so MSI installs work without registry workaround +- [ ] GuruRMM Issue #9: Long-term fix for rmm-api Cloudflare proxy (tunnel or proper WebSocket proxy config) +- [ ] Link enrolled_agents record (bf87301e) to agent record (7626d82c) in DB — currently agent_id = NULL +- [ ] Western Tire ticket #32199: Bill when scope confirmed +- [ ] westerntire.com SSL cert expires 2026-05-30 — verify AutoSSL will renew +- [ ] Western Tire Syncro customer record: Update "DNS Detail" field (still says "Email is on Websvr") +- [ ] Syncro ticket #32185 (Dataforth): Duplicate comment needs manual GUI deletion (from prior session) + +--- + +## Technical Notes + +### GuruRMM enrolled_agents Workaround +The standard MSI install flow writes SiteId to registry, agent enrolls and gets AgentKey. BUT the WebSocket auth in ws/mod.rs never checks enrolled_agents — it only validates against the `sites` table (site-based) or `agents` table (legacy). The AgentKey from enrollment is useless for WebSocket auth. + +**Workaround for any new Windows agent installs until Issue #8 is fixed:** +After install, overwrite registry AgentKey with the site API key: +```powershell +Set-ItemProperty -Path 'HKLM:\SOFTWARE\GuruRMM' -Name 'AgentKey' -Value '' -Type String +Restart-Service GuruRMMAgent -Force +``` +Site API keys are in the GuruRMM DB `sites.api_key` column or in the vault per-site. + +### Syncro Comment jq Path Bug (fixed in syncro.md) +POST /comment response is `{"comment": {...}}` not a flat object. +- Wrong: `| jq '{id: .id}'` → returns null, looks like failure +- Correct: `| jq '{id: .comment.id}'` +This caused duplicate public comments on tickets #32185 (Apr 22) and #32142 (Apr 23). +syncro.md updated with correct pattern and example code. + +### Dataforth Prepaid Hours +All Dataforth invoices are $0.00 — they are on a prepaid hours block (monthly $2,098.87 replenishment invoices). Labor line items consume from the block rather than generating new dollar charges. 2 hours consumed today for ticket #32142.