diff --git a/.claude/commands/syncro-emergency-billing.md b/.claude/commands/syncro-emergency-billing.md index b449ee2..4397c27 100644 --- a/.claude/commands/syncro-emergency-billing.md +++ b/.claude/commands/syncro-emergency-billing.md @@ -75,7 +75,8 @@ Use a **single emergency labor product** for the full hours worked: | Customer type | Product | Product ID | Rate | |---|---|---|---| | Business | Labor - Emergency or After Hours Business | 26184 | $262.50/hr | -| Residential | Labor - Emergency or After Hours Residential | 42584 | $202.50/hr | + +Residential rates are legacy — ACG no longer bills residential. Do not use product 42584. **Example:** 4 hours of emergency business work, direct billing: - Line 1: `Labor - Emergency or After Hours Business` (26184) — 4.0 hr @@ -86,15 +87,52 @@ Reference: Ticket #32188 (VWP, direct billing, 2026-04-22). ## Standard Labor Products (Reference) -Standard non-block business rate: **$175/hr**. Emergency premium is 50%, so $175 × 1.5 = $262.50 (product 26184). +Emergency business rate is **$262.50/hr** (product 26184) — used for all emergency/afterhours business work regardless of remote vs onsite. Residential rates are legacy and not in use. -| Service type | Product | Product ID | Rate | Notes | +**Always fetch `price_retail` from `GET /api/v1/products/{id}` before billing non-block customers. Never use a hardcoded rate.** + +| Service type | Product | Product ID | Live Rate | Notes | |---|---|---|---|---| -| Remote Business | Labor - Remote Business | 1190473 | $175.00/hr | Non-block cash billing | -| Onsite Business | Labor - Onsite Business | 26118 | $175.00/hr | Same rate as remote | -| In-Shop Business | Labor - In Shop Business | 573881 | $175.00/hr | | +| Remote Business | Labor - Remote Business | 1190473 | $150.00/hr | Non-block cash billing | +| Onsite Business | Labor - Onsite Business | 26118 | $175.00/hr | | +| In-Shop Business | Labor - In Shop Business | 573881 | $150.00/hr | | | Block/Prepaid (any type) | Labor - Remote Business | 1190473 | $0.00 | Price = $0; draws from block in hours | -| Remote Residential | Labor - Remote Residential | 1190471 | $100.00/hr | | +| Emergency/Afterhours Business | Labor - Emergency or After Hours Business | 26184 | $262.50/hr | All business emergency — remote and onsite | + +--- + +## Adding Line Items to an Existing Ticket (API) + +**Confirmed working endpoint** (tested 2026-05-25, ticket #32320 and test #32321): + +``` +POST https://computerguru.syncromsp.com/api/v1/tickets/{ticket_id}/add_line_item +Authorization: +Content-Type: application/json +``` + +**Required body fields** — both `name` and `description` are required; either missing returns 422: + +```json +{ + "product_id": 1190473, + "name": "Labor - Remote Business", + "description": "Work performed description", + "quantity": 2.0, + "price": 0.0, + "taxable": false +} +``` + +- `price`: use `0.0` for block customers; for non-block, fetch live rate first: `GET /api/v1/products/{product_id}` → `.product.price_retail` +- `taxable`: always `false` for labor (Arizona labor is never taxable) +- Success response: HTTP 200 with the new line item's `id` + +**Pre-billing check** — before adding line items, verify the ticket has no existing labor to avoid duplicates: +``` +GET https://computerguru.syncromsp.com/api/v1/tickets/{ticket_id} +``` +Check `.ticket.line_items[]` in the response. --- @@ -113,6 +151,8 @@ If work begins during normal hours and continues into after-hours (or vice versa | LLF - Remote Labor (Emergency/Afterhours) | 145022 | Legacy contract product, no longer applicable | | Fee - On-Site Business Emergenc | 45871 | Not in current use — do not add without explicit instruction | | Fee - On-Site Residential Emerg | 45870 | Not in current use — do not add without explicit instruction | +| Labor - Emergency or After Hours Residential | 42584 | Residential rates are legacy — ACG no longer uses residential billing | +| Labor - Remote Residential | 1190471 | Residential rates are legacy — ACG no longer uses residential billing | --- @@ -129,5 +169,5 @@ Was emergency rate explicitly requested by customer? ``` Is customer a block/prepaid customer? YES → Two line items: actual hrs (standard product) + 0.5x hrs (same product, "Emergency/Same day rate") - NO → One line item: emergency product (26184 business / 42584 residential) for full hours + NO → One line item: emergency product 26184 (business) for full hours ``` diff --git a/session-logs/2026-05-25-session.md b/session-logs/2026-05-25-session.md index cc26b25..112684b 100644 --- a/session-logs/2026-05-25-session.md +++ b/session-logs/2026-05-25-session.md @@ -1088,3 +1088,115 @@ if let (Some(version), Some(arch)) = ( - 12:25 PT - Final compilation successful on Saturn - 12:40 PT - Session log written, ready to sync + +--- + +## Update: 12:55 PT — Dataforth ESXi License Recovery + Syncro Emergency Billing Skill + +### User +- **User:** Mike Swanson (mike) +- **Machine:** GURU-5070 +- **Role:** admin +- **Session span:** ~2026-05-24 evening – 2026-05-25 afternoon + +### Session Summary + +Session began as an emergency response: John Lehman texted after hours reporting VPN was down. Investigation via SSH (through D2TESTNAS at 192.168.0.9 as jump host) revealed AD1 and AD2 were offline because ESXi-122's 60-day evaluation license had expired, taking all VMs with it. ESXi-124 was also at risk. SSH was not running on ESXi-122, requiring DCUI physical console access to enable it first. + +License recovery on ESXi-122 was accomplished by copying the hidden backup license file (`/etc/vmware/.#license.cfg`) over the active `license.cfg`, then restarting hostd. This resets the 60-day evaluation timer. ESXi-124 was treated preemptively with the same procedure. After license restoration, all four VMs on ESXi-122 (AD1, AD2, FILES-D1, PBX) were powered on. Both ESXi hosts were configured with a persistent monthly cron job (first Sunday of each month at 02:00) to auto-reset the license and reboot, written directly to `/var/spool/cron/crontabs/root` via paramiko SFTP and persisted through `/etc/rc.local.d/local.sh` since ESXi's filesystem is RAM-based. + +A Syncro ticket was created (#32320) for the incident. The session then shifted to building out emergency/afterhours billing rules as a skill file (`syncro-emergency-billing.md`), researching Winter's historical tickets to establish the correct billing pattern. The key finding: block customers (Dataforth, VWP, Cascades) require two line items on the standard product (actual hours + 0.5x labeled "Afterhours rate") because block accounts track hours not dollars; non-block customers use a single dedicated emergency product (26184, $262.50/hr). + +Adding labor to the Dataforth ticket required discovering the correct Syncro API endpoint through trial and error — `/tickets/{id}/add_line_item` (not `/line_item`, `/line_items`, or top-level endpoints). Experimented on ACG internal test ticket #32321 to confirm payload format before touching the real ticket. Once confirmed, added 2.0hr main labor + 1.0hr afterhours premium to ticket #32320, then deleted the test ticket. The skill was then audited: live product rate fetch revealed two rate errors in the original draft ($150/hr not $175 for Remote Business and In-Shop Business), residential rates were removed as legacy, and the confirmed API method was documented with all required fields. + +### Key Decisions + +- **ESXi crontab via SFTP, not shell**: ESXi has no `crontab` command. Wrote directly to `/var/spool/cron/crontabs/root` via paramiko SFTP; sent SIGHUP to crond after. Shell-based approaches (echo/heredoc) were tried first and failed. +- **local.sh persistence in Python, not shell**: `grep -c` through a shell command produced "0\n0" (grep output + fallback), causing false-positive match detection. Rewrote local.sh update logic using SFTP read/write in Python to avoid shell quoting/output ambiguity. +- **Test before touching real ticket**: Rather than guessing the Syncro line item payload format and hitting the real Dataforth ticket, opened a test ticket on ACG internal customer to confirm endpoint and required fields first. +- **Both `name` and `description` required**: Syncro's `add_line_item` endpoint returns 422 if either field is missing — not obvious from the API name. Documented explicitly. +- **Live rate fetch mandatory**: Memory note confirmed rates had been wrong before (2026-05-20 incident). Fetched all product rates live before finalizing the skill; found Remote Business ($150) and In-Shop Business ($150) were both documented as $175 in the original draft. +- **$262.50 emergency product covers all business work**: Confirmed with Mike — no distinction between remote and onsite emergency. One product for all business emergency billing regardless of service delivery method. +- **Residential rates are legacy**: Removed 42584 and 1190471 from all active sections of the skill; added to "Products NOT to Use." + +### Problems Encountered + +- **SSH not enabled on ESXi-122**: License expiration locks out management — had to enable SSH via DCUI physical console before remote work was possible. No automated fix; required hands-on at the host. +- **`crontab` command missing on ESXi**: ESXi busybox environment does not include the `crontab` CLI. Fix: write the crontab file directly via SFTP. +- **`grep -c` false positive in local.sh check**: Shell command `grep -c 'pattern' file 2>/dev/null || echo 0` emitted both the grep count and the fallback "0", causing the Python string comparison to see "0\n0" (truthy). Fixed by using SFTP to read and rewrite local.sh entirely in Python. +- **Syncro line item endpoint discovery**: No working documentation for the correct path. Tried `/line_item`, `/line_items`, PUT with `line_items_attributes` — all 404. Eventually fetched the Syncro Swagger spec from `api-docs.syncromsp.com/swagger.json` and found `add_line_item`. +- **422 on add_line_item with only `name` field**: Both `name` and `description` are required; omitting either returns 422. + +### Configuration Changes + +- **Created:** `D:\claudetools\.claude\commands\syncro-emergency-billing.md` — Emergency/afterhours billing skill for Syncro (rules, billing scenarios, confirmed API method) +- **Modified:** `syncro-emergency-billing.md` — Rate corrections (Remote Business $150, In-Shop $150), residential removed as legacy, API section added +- **ESXi-122** (`192.168.0.122`): license.cfg restored, cron job written, local.sh updated, all VMs powered on +- **ESXi-124** (`192.168.0.124`): license.cfg restored preemptively, cron job written, local.sh updated + +### Credentials & Secrets + +- **D2TESTNAS (jump host):** `192.168.0.9` — root / `Paper123!@#` +- **ESXi root password (both hosts):** `Gptf*77ttb!@#!@#` +- **Syncro API key:** `T259810e5c9917386b-52c2aeea7cdb5ff41c6685a73cebbeb3` — vault: `msp-tools/syncro.sops.yaml` → `credentials.credential` + +### Infrastructure & Servers + +| Host | IP | Role | Notes | +|---|---|---|---| +| D2TESTNAS | 192.168.0.9 | Jump host / NAS | SSH root access; used as paramiko jump for ESXi | +| ESXi-122 | 192.168.0.122 | Hypervisor | Datastore: `datastore1`; hosts AD1, AD2, FILES-D1, PBX | +| ESXi-124 | 192.168.0.124 | Hypervisor | Datastore: `Backup`; treated preemptively | +| AD1 | (on ESXi-122) | Domain Controller | Was offline due to license expiry; restored | +| AD2 | (on ESXi-122) | Domain Controller | Was offline; restored | +| FILES-D1 | (on ESXi-122) | File server | Was offline; restored | +| PBX | (on ESXi-122) | Phone system | Was offline; restored | + +ESXi license reset script locations: +- ESXi-122: `/vmfs/volumes/datastore1/license_reset.sh` +- ESXi-124: `/vmfs/volumes/Backup/license_reset.sh` + +Cron schedule (both hosts): `0 2 * * 0 [ $(date +%d) -le 7 ] &&