claudetools/session-logs/2026-05-01-howard-syncro-billing-batch-and-tmp-path-incident.md

# 2026-05-01 — Howard — Syncro billing batch + /tmp path mismatch incident

## User
- **User:** Howard Enos (howard)
- **Machine:** Howard-Home
- **Role:** tech

## Session Summary

Billed three Syncro tickets in one sitting: an onsite computer-replacement job for Sombra Residential, a 1-hour emergency BSOD diagnosis for Mineralogical Record, and the multi-day Cascades of Tucson Entra setup project. Each ticket got a Resolution comment, a billable line item, and a Standard invoice. Cascades was billed against its 33.5-hour prepaid block at $0/hr; the other two were standard cash invoices.

The Sombra ticket triggered a real incident: the first comment that landed on the ticket had completely wrong content — a body about "Karen Rossini" and the ALDOCS share at Cascades, totally unrelated to Sombra. Investigation found the root cause was a Windows path-resolution mismatch. The Claude Code Write tool resolves `/tmp` to `C:\tmp\` while Git Bash + curl resolve it to `%LOCALAPPDATA%\Temp\` — two different real directories. The Write tool reported success, but the file landed in `C:\tmp\` and curl read a stale payload from yesterday's Cascades session at the bash-side `/tmp`. Howard manually deleted the rogue comment in Syncro (no API delete exists for comments). Subsequent posts on all three tickets used heredoc payloads piped into curl directly — no file handoff, no path ambiguity, no further problems.

After the incident, root cause was saved to `.claude/memory/feedback_tmp_path_windows.md` and indexed in `MEMORY.md`. A one-time remote agent was scheduled to fire 2026-05-02 08:00 PDT to update the syncro skill's examples to use heredocs instead of `/tmp` files (so the next person on a Windows machine doesn't trip the same wire).

## Key Decisions

- **Sombra resolution comment kept hidden (internal-only)** — passwords were documented in the body. Customer-visible would have emailed plaintext credentials to Rishi. Hidden + `do_not_email: true` was the safer default.
- **Cascades billing — 33.5 hrs project labor only, no overage** — Howard's call. Winter (front desk / billing) will bill Cascades for a fresh prepaid block separately rather than rolling overage into this ticket.
- **Cascades pre-4/24 work scoped to Entra-relevant items only** — folder redirection, email security/DMARC, Synology DSM discovery, and Jeff Bristol mailbox restore (~9 hrs) were excluded from billing on this ticket because they likely belong on other Cascades tickets.
- **Mike's audit retention design (~2-3 hrs) and onboard-tenant.sh patches (~1 hr) excluded** — partly internal architecture work that benefits all future tenants, not solely Cascades.
- **Cascades comment split into two** — long detailed 4/25-4/30 narrative kept hidden; short status summary posted public so the customer gets a readable update. Hidden comment on internal work, public comment on customer-facing what's-done / what's-next.
- **Cascades ticket left In Progress (not Invoiced)** — Syncro auto-flipped to Invoiced when the $0 invoice posted; reverted to In Progress per Howard's instruction. Project ongoing.
- **Mineralogical Record emergency = product 26184 (no quantity multiplier)** — verified `prepay_hours: "0.0"` first. Non-prepaid customers get `26184` at actual hours; the 1.5× emergency multiplier is baked into the rate. No quantity bump needed.

## Problems Encountered

- **Wrong-content comment posted to Sombra ticket #32225** — Comment ID #408671678 contained a Karen Rossini / ALDOCS body. Caused by `/tmp` resolving to two different real directories on Windows. Resolution: Howard manually deleted the comment in the Syncro GUI (API has no DELETE for comments). All subsequent POSTs in this session used heredoc payloads. Root cause saved to `feedback_tmp_path_windows.md`.
- **Cascades prepaid balance did not auto-debit on $0 invoice creation** — After posting invoice #67537 with 33.5 hrs of product `9269129` Project Labor at $0, the customer's `prepay_hours` field still shows `33.5`. Hypothesis: the debit fires when the invoice is paid/posted in the Syncro GUI, not on creation. **Action for Winter** — verify the debit lands when she handles billing for the new prepaid block.
- **Cascades ticket auto-flipped to Invoiced** — Syncro flips status to Invoiced automatically when an invoice is generated against a ticket. Reverted to In Progress with a `PUT /tickets/{id}` after the invoice posted.

## Tickets Worked

### Ticket #32225 — Sombra Residential — Onsite computer setup (2 machines)
- **Customer ID:** 32971820
- **Customer prepay_hours:** none
- **Resolution comment ID:** #408673155 (hidden=true, do_not_email=true)
  - Documented: 2 new computers deployed (front desk, Bryan's), profiles/data migrated, Windows password set to `Sombra11225`, localadmin account created with password `$ombr@11225`, M365 sign-in verified, printers reconnected, network shares verified
  - Note re Bryan's old machine: ~250GB of data sits in C:\ root; per Rishi, user will move it himself if wanted (would have added ~4 hrs of onsite time)
- **Line item:** #42259906 — product `26118` Onsite Business — 3.0 hrs × $175.00 = $525.00, taxable=false
- **Invoice:** #67535 — total $525.00
- **Status:** Invoiced
- **Rogue/deleted comment:** #408671678 (manually deleted by Howard from GUI)

### Ticket #32229 — Mineralogical Record — Emergency BSOD onsite
- **Customer ID:** 207770
- **Customer prepay_hours:** 0.0 (verified before applying emergency rate)
- **Resolution comment ID:** #408674727 (hidden=false, do_not_email=false — public, customer emailed)
  - Diagnosed Intel wireless driver as cause of BSOD 0xD1 from crash dump logs
  - Updated Intel wireless driver to latest version
  - Customer running overnight memtest, will follow up if issues
- **Line item:** #42260143 — product `26184` Emergency or After Hours Business — 1.0 hr × $262.50 = $262.50, taxable=false
- **Invoice:** #67536 — total $262.50
- **Status:** Invoiced

### Ticket #32214 — Cascades of Tucson — Entra setup (project)
- **Customer ID:** 20149445
- **Customer prepay_hours BEFORE invoice:** 33.5
- **Customer prepay_hours AFTER invoice:** 33.5 (debit deferred — see Problems Encountered)
- **Private comment ID:** #408677784 (hidden=true) — full date-grouped narrative for 2026-04-25 through 2026-04-30 covering Entra Connect staging install, CA reconciliation, Phase B Intune completion, audit retention design approval, pilot user + 3 CA policies (Report-only), MHS kiosk fix (package names not GUIDs), SDM bootstrap in flight, Tenant Admin SP scope expansion to 14 Graph application roles
- **Public comment ID:** #408677808 (hidden=false, do_not_email=false) — short customer-facing status (what's done / what's next)
- **Line item:** #42260540 — product `9269129` Prepaid Project Labor — 33.5 hrs × $0.00 = $0.00, taxable=false
- **Invoice:** #67537 — total $0.00
- **Status:** In Progress (manually reverted from auto-flipped Invoiced)
- **Excluded from billing on this ticket:** folder redirection (~2), email security + DMARC (~4), Synology DSM (~2), Jeff Bristol mailbox restore (~1), Mike audit retention design (~2-3), onboard-tenant.sh patches (~1) — total ~12-13 hrs not billed here

## Heads-up for Winter

**Cascades of Tucson — Entra setup ticket #32214:**
- Invoice #67537 ($0.00) posted today against the 33.5-hour prepaid block
- The customer's `prepay_hours` field still shows 33.5 — debit may not fire until the invoice is paid/posted in the Syncro GUI. Please verify the bank zeroes out as expected.
- Cascades needs a fresh prepaid block billed (separate transaction). Howard's intent: zero out the existing block via this $0 invoice, then you bill them for a new block of prepaid hours.
- Project is ongoing — ticket left In Progress, more line items will land on it as work continues.

## Credentials
None used or discovered this session beyond the per-user Syncro API tokens, which are documented in the syncro skill itself and vaulted under `msp-tools/syncro-howard.sops.yaml`.

Howard's Syncro API token (per-user, attribution = Howard Enos, user_id 1750):
- `Tde5174a6e9e312d14-02fd5bfe0f0ee40c87d027507c680e18`

For Sombra Residential machines deployed today (now in production, customer responsibility):
- Windows user password (both machines): `Sombra11225`
- localadmin account password (both machines): `$ombr@11225`

## Infrastructure & Servers
No infrastructure changes this session. Endpoints touched:
- Syncro API base: `https://computerguru.syncromsp.com/api/v1`
- Tickets: 109655876 (Sombra), 109708907 (Mineralogical Record), 109412123 (Cascades)
- Customers: 32971820 (Sombra), 207770 (Mineralogical Record), 20149445 (Cascades)
- Invoices: 67535 ($525), 67536 ($262.50), 67537 ($0)
- Line items: 42259906, 42260143, 42260540
- Comments posted: 408673155 (Sombra res), 408674727 (Mineralogical res), 408677784 (Cascades private), 408677808 (Cascades public)
- Comments deleted (manual GUI): 408671678 (Sombra rogue)

## Commands & Outputs

All Syncro POSTs in the second half of the session used heredoc payloads, e.g.:

```bash
curl -s -X POST "${BASE}/tickets/${TICKET_ID}/comment?api_key=${API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary @- <<'JSON'
{"subject": "...", "body": "...", "hidden": false, "do_not_email": false}
JSON
```

Path resolution diagnostic that confirmed root cause:
```
$ cd /tmp && pwd -W
C:/Users/Howard/AppData/Local/Temp     # Git Bash sees /tmp here

$ ls -la "C:/tmp/comment_payload.json"
-rw-r--r-- 1 Howard 197609 814 May  1 07:10  # Write tool wrote here

$ ls -la "/c/Users/Howard/AppData/Local/Temp/comment_payload.json"
-rw-r--r-- 1 Howard 197609 240 Apr 30 16:51  # Stale file curl actually read
```

## Configuration Changes

### Files created
- `C:\claudetools\.claude\memory\feedback_tmp_path_windows.md` — root cause writeup + recommended fix patterns

### Files modified
- `C:\claudetools\.claude\memory\MEMORY.md` — added pointer entry for the new feedback memory

## Scheduled Work

**Routine ID:** `trig_01CAfvwoQ4nLcKEqbU4UQmSa`
- Fires once: 2026-05-02 08:00 PDT (2026-05-02T15:00:00Z)
- Model: claude-sonnet-4-6
- Repo: claudetools
- Task: Update the syncro skill's examples to use heredoc payloads (`curl --data-binary @- <<'JSON' ... JSON`) instead of `/tmp/<file>.json` paths, then commit and push to `main`. Skill should not change rate tables or logic — only the payload-handoff mechanic.
- Manage: https://claude.ai/code/routines/trig_01CAfvwoQ4nLcKEqbU4UQmSa

## Pending / Incomplete Tasks

- Verify Cascades `prepay_hours` debits to 0 after invoice #67537 settles (Winter)
- Winter to bill Cascades for a new prepaid block
- Scheduled agent (above) to update the syncro skill examples — fires 2026-05-02 08:00 PDT
- Cascades Entra project work continues — Entra Connect staging exit, CA enforce flip, remaining caregiver phone enrollments, audit retention infra build, break-glass admin account creation (all tracked in PROJECT_STATE.md)

## Reference

- Syncro skill: `.claude/commands/syncro.md`
- Syncro per-user tokens: `msp-tools/syncro-howard.sops.yaml`, `msp-tools/syncro-mike.sops.yaml`
- Cascades client folder: `clients/cascades-tucson/`
- Cascades project state: `clients/cascades-tucson/PROJECT_STATE.md`
- New memory: `.claude/memory/feedback_tmp_path_windows.md`

---

## Update: 20:07 PDT — Time-tracking workflow finding + skill rewrite

After the billing work above, ran `/sync` and pulled in 5 commits from Mike. Mike's 4/30 session log contained a `## Note for Howard` flagging that **all 31 closed tickets he audited had 00:00:00 in Syncro's time-tracking system** — invoices were created with bare `add_line_item` calls and hours were typed into description text. This bypasses Syncro's time-entries table entirely and breaks reporting (hours per client, technician productivity, average resolution time, prepay burn rate).

The required workflow per Mike: `timer_entry → charge_timer_entry → invoice`. Bare `add_line_item` is only acceptable for non-time items (hardware, flat-fee services with zero labor). Even warranty/free work needs a time entry (`billable: false`); only cancelled tickets are exempt.

I had just done exactly the wrong thing on three tickets earlier in this session. Winter retroactively added time entries to fix #32225, #32229, and #32214.

### Actions taken on this finding

1. **Saved feedback memory** at `.claude/memory/feedback_syncro_timer_first.md` documenting the rule, the rationale, and the two-day repeat-incident timeline (Mike caught 31 tickets 4/30; I repeated on 3 more 5/01).
2. **Saved a separate feedback memory** at `.claude/memory/feedback_syncro_cascades_contact.md` for an unrelated rule Howard surfaced mid-session: never set `contact_id` on Cascades tickets — leaving it blank lets Syncro route to the correct email distribution; setting it (Meredith Kuhn has been the recurring incorrect default) overrides and breaks the distribution.
3. **Delegated a syncro skill rewrite** to the Coding Agent. Result committed as `ec98c6c` and pushed:
   - Promoted `timer_entry → charge_timer_entry` to the documented default `/syncro bill` flow (was 13 steps using `add_line_item`; now 16 steps using the timer path).
   - Demoted bare `add_line_item` to a clearly-labeled fallback section for non-time items only. Example payload there is now a hardware item, not labor.
   - Added two new Hard Rules at the top of the skill: timer-entry-first rule (with warranty exception) and heredoc-payload rule (no `/tmp/*.json` files).
   - Replaced every `/tmp/*.json` write+curl pattern with `--data-binary @- <<'JSON' ... JSON` heredocs. Quoting strategy: `<<'JSON'` (literal) for static payloads, `<<JSON` (interpolating) for payloads that need shell variables like `${TIMER_ID}`.
   - Updated Ollama drafting block to use `$CLAUDETOOLS_ROOT/.claude/tmp/ollama_prompt.txt` (workspace path both Write and Bash agree on) instead of `/tmp/ollama_prompt.txt`.
   - Updated `/syncro bill <number>` arg list to include `warranty` as a labor type.
   - Preserved emergency-billing prepaid branching (product 26184 vs 26118 × 1.5), per-user attribution rules, comment-shape warnings, rate tables, endpoint reference tables, and Cascades contact rule references — all untouched.
4. **Disabled scheduled remote agent** `trig_01CAfvwoQ4nLcKEqbU4UQmSa` (was set to fire 2026-05-02 08:00 PDT to do the heredoc-only fix). Its scope was a strict subset of the larger rewrite that just shipped, so leaving it enabled would have been a no-op at best. Updated to `enabled: false` via RemoteTrigger.

### Files changed in the second half of session

- `.claude/commands/syncro.md` — major rewrite (timer-first + heredoc throughout)
- `.claude/memory/MEMORY.md` — index entries for the two new memories
- `.claude/memory/feedback_syncro_timer_first.md` (new)
- `.claude/memory/feedback_syncro_cascades_contact.md` (new)

### Repo state at end of session

- HEAD: `ec98c6c syncro skill: timer-entry-first workflow + heredoc payloads`
- Local + origin/main in sync
- Three Howard-authored commits today: session log + sync auto-commit + skill rewrite
- Three new feedback memories landed today (tmp-path, Cascades-contact, syncro-timer-first)
- One disabled remote routine (no longer needed)

### Pattern to apply going forward

Every Syncro billing operation in future sessions must be:
1. POST `/tickets/{id}/timer_entry` (with `start_at`, `end_at`, `billable`, `product_id`, `notes`) → record time
2. POST `/tickets/{id}/charge_timer_entry` (with `{"timer_entry_id": N}`) → auto-generate line item
3. Verify line item came in with correct `price_retail`; PUT-patch via `update_line_item` if it landed at $0
4. POST `/invoices` → roll line item onto invoice
5. PUT ticket status to `Invoiced`

The skill now reflects this. The memory now reinforces it. If a future session reverts to bare `add_line_item` for time-bearing work, it's a regression — call it out and use the timer path.