Auth selection logic:
- Default: prefer cert when cert_thumbprint_b64url + cert_private_key_pem_b64
are present in the vault entry's credentials block; fall back to client_secret.
- REMEDIATION_AUTH=secret -> force client_secret flow.
- REMEDIATION_AUTH=cert -> force cert flow; error if cert fields missing.
- Logs [INFO] auth=cert/secret to stderr so users see which path was taken.
Cert flow signs an RS256 JWT (header includes x5t) via inline Python (PyJWT
+ cryptography), POSTs client_assertion_type +
client_assertion=<jwt> in place of client_secret. Same scope, same cache, same
error handling (AADSTS7000229 still emits the consent URL).
Single sops -d to a mktemp file feeds both field reads to avoid repeated
~1s decrypt invocations on Windows; trap removes plaintext on exit.
Verified end-to-end against tedards.net for all three modes after wiping
/tmp/remediation-tool/.
- Promote timer_entry → charge_timer_entry to default billing path; demote
bare add_line_item to a clearly-labeled fallback for non-time items only.
Mike caught the bare-add_line_item bug across 31 tickets on 2026-04-30;
repeated on 3 tickets 2026-05-01. Time entries are required for Syncro
reporting (hours per client, tech productivity, prepay burn).
- Replace /tmp/*.json payload pattern with heredoc throughout. /tmp resolves
to C:\tmp\ in the Write tool but %LOCALAPPDATA%\Temp\ in Git Bash on
Windows — different real directories. Caused a wrong-comment incident on
ticket #32225 2026-05-01 (rogue payload from prior session). Heredoc
avoids the file handoff entirely.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three tickets billed today: #32225 Sombra ($525 onsite), #32229 Mineralogical
Record ($262.50 emergency), #32214 Cascades Entra (33.5 hrs project labor at $0
debits prepaid block). Hit a real incident on Sombra: rogue comment posted with
content from a different ticket because /tmp resolves differently in the Write
tool (C:/tmp/) vs Git Bash (%LOCALAPPDATA%/Temp/) on Windows. Howard manually
deleted from GUI; subsequent posts used heredoc to avoid the file handoff
entirely. Root cause documented in feedback_tmp_path_windows.md so future
sessions don't trip the same wire. Scheduled remote agent
trig_01CAfvwoQ4nLcKEqbU4UQmSa to update the syncro skill examples 2026-05-02.
Frontend pass on the two embedded HTML templates in the FastAPI server. No
backend / Python logic changed; only template strings, CSS, and inline JS.
Index page: full CSS custom-property theme (light, #c39733 accent),
responsive viewport meta, search input with embedded SVG magnifier and
focus ring, control bar reorganised into divider-separated groups with
the browse-mode toggle rendered via :has() selector, hit cards with
hover-lift + arrow indicator and focus-visible outline, restyled Q/A
badges and score/topic chips, animated loading dots.
Episode page: sticky audio player and sticky aside (top: 130px,
max-height calc'd against viewport). New active-Q&A highlight builds a
sorted index of QA blocks at load time, computes each block's end as
the next block's start (capped at +180s), and on timeupdate/pause
toggles .active on both the body QA block and its aside list item; a
"NOW PLAYING" pill is revealed on .qa.active. Intro-marker also gets
.active. Audio preload bumped from none to metadata so #qa-<id> deep
links can seek without a prior user gesture.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Updated Howard's note with correct analysis after Mike's clarification:
BUSINESS RULE (from Mike):
- ALL tickets need time entries (except cancelled)
- Even warranty/free work logs time
- Time tracking separate from billing decisions
FINDINGS:
- Billing: ✅ Working (29 invoices exist, 2 correctly non-billed)
- Time tracking: ❌ Bypassed (all 31 show 00:00:00)
ROOT CAUSE:
- Manual invoice line items used instead of time tracking
- Hours typed in descriptions ("Applied X.0 Prepay Hours")
- Prevents productivity/utilization reporting
Pattern: 20 prepay deductions + 16 direct charges, all via manual
line items. Workflow skips Syncro time tracking system entirely.
Examples included with hours that should have been logged.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Created memory entry documenting correct way to verify ticket-invoice linkage
in Syncro API after 2026-04-30 incident where faulty verification script
falsely claimed 31 tickets had no invoices (actually 29 had invoices properly,
2 were correctly Non-Billable).
Key lessons:
- List endpoint does NOT return ticket_id or line_items
- Must query individual invoices for full data
- Invoice numbers are strings, not integers
- Use ticket ID (internal), not ticket number (user-visible)
Added to memory index for future GrepAI semantic search.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Previous commits falsely claimed 31 tickets had no invoices. This was based on
a fundamentally flawed verification script that:
- Used list endpoint instead of individual invoice details
- Failed to check invoice-level ticket_id field
- Had type comparison errors (string vs int)
CORRECTED FACTS:
- 29 out of 31 tickets DO have proper invoices (93.5% success)
- 2 tickets correctly have no invoices (marked Non-Billable)
- #32083 (DAnaise.com): Non-Billable status
- #32022 (Michael Johnson): Cancelled, Non-Billable
NO ACTION REQUIRED - Howard's billing workflow is working correctly.
Sincere apologies for the false alarm. Mike caught the error immediately.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Pattern analysis reveals:
- 31 tickets span March 3 - April 28 (not one-time event)
- Multiple update date clusters (batch processing pattern)
- All missing normal invoice workflow steps
- Tickets changed to 'Invoiced' status without:
* Time entries
* Invoice generation
* Workflow comments
NOT a Claude/API integration issue - Claude doesn't change ticket statuses.
Likely causes:
1. Manual bulk status updates to clear queue
2. Misconfigured Syncro automation/workflow
3. Periodic batch status changes
Urgent: Need to review Syncro automation rules and prevent future revenue loss
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Deep verification performed:
- Checked customer invoice records for all 31 tickets
- ZERO invoices found matching these tickets
- Cascades confirmed to have NO contract (11 tickets affected)
- Example: Kittle #32223 marked 'Invoiced' but no invoice exists
- This is genuine lost revenue, not contract-covered work
Estimated impact: 31 billable tickets with no revenue captured
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Major billing gap identified:
- 39 tickets closed/invoiced today
- 31 have ZERO time logged (00:00:00)
- Many marked 'Invoiced' but sent with no time
- Detailed list provided for review and correction
Sombra RMM enrollment: no billing needed per Mike
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Ticket #32225 exists but has no time logged
- Today's GuruRMM enrollment work is unbilled
- Needs either ticket update or new ticket creation
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Cascades caregiver shared-phone bypass pilot — 2026-04-29 evening into
2026-04-30 early morning continuation.
Major work:
- Adopted phased per-group CA rollout (corrects original tenant-wide §5
design that would have blocked off-site office users)
- Step A: backfilled admin@ into excludeUsers on all 8 existing Cascades
CA policies (mirrors sysadmin@ exclusion posture; Option 1 break-glass)
- Outlook + Helpany + LinkRx assigned to Cascades - Shared Phones group
and added to MHS kiosk app list (final dashboard: 5 caregiver apps)
- Created cloud-only pilot user pilot.test@cascadestucson.com,
SG-Caregivers-Pilot group, Business Premium license, vault entry
pushed to Gitea vault repo
- Built 4 CA changes: PATCH legacy all-users-MFA to exclude pilot group,
CREATE 3 new Report-only policies (block off-network, block
non-compliant, 8h sign-in frequency) with both admins excluded
- Pilot phone wipe + re-enroll after first attempt stuck; PIN set,
awaiting MHS to take over launcher and SDM sign-in prompt
6 new project/feedback memories. Resume point at top of new session log.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both servers were already patched (11.110.0.97 and 11.134.0.20) via
daily auto-update. IOC scan found 16 flagged sessions across both
plus 4 uncommented SSH keys on IX.
Critical remediation:
- Forensic evidence preserved before any deletion
- 4 uncommented SSH keys removed from IX (server-side backup retained)
- 16 flagged sessions purged across both servers
- Root passwords rotated via chpasswd
- New WHM API tokens created; 3 stale transfer-* tokens revoked
- Vault entries + 1Password Infrastructure items updated
Forensic deep-dive verdict: patch held. All 7 actual CVE exploit
attempts (botnet IPs hitting /json-api/version) returned HTTP 403.
The "multi-line pass" IOC hits on user sessions were false positives.
Unidentified 76.18.103.222 root session traced to routine SSL
maintenance (zero sensitive endpoints touched).
Skill hardening:
- Added MANDATORY service-token directive to .claude/commands/1password.md
enforcing OP_SERVICE_ACCOUNT_TOKEN from SOPS for all op CLI calls
- Per Mike: memory files alone don't reliably bind agent behavior;
baking governance into skill content loaded at moment of use.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make the radio archive Q&A pairs actually browseable end to end:
- /api/qa list endpoint (year, min_score, exclude_banter, topic_class,
pagination, sort by air_date or score). Returns the same column shape as
/api/search Q&A hits.
- /api/audio/{episode_id} streams the MP3 with HTTP Range support so the
browser <audio> can seek. 206 + Content-Range when ranged, 200 when
full-file. Returns 404 cleanly when episodes/ tree is absent (Jupiter).
- /episode/{id} HTML transcript view: chronological segments with clickable
timestamps, Q&A blocks spliced inline (anchor #qa-<id>), intros marked
inline, right-rail summary. Hash-anchor on load auto-seeks the audio.
- New question_excerpt / answer_excerpt fields on /api/search Q&A hits and
on /api/qa items: trim leading run-on chatter, take ~300 chars, end on a
sentence boundary or word boundary with ellipsis.
- Index UI: each Q&A hit now links to /episode/{id}#qa-{qa_id}; new
"Browse all Q&A" toggle (year selector, sort, append-load 50 per page,
defaults to min_score=3); FTS snippet replaced with the plain excerpt
when available.
No new dependencies, no schema changes, no LLM calls. Uses
EPISODES_DIR env (default /data/episodes) — Jupiter compose still only
mounts /data so audio degrades gracefully to 404 there until episodes
are uploaded.
Backend min_score/exclude_banter wired through to HTML index. Adds
score badges (1-5 red->green), topic_class pills, dim styling on
banter rows. Live on http://172.16.3.20:8765/. Synced to portable
repo. pscp ENOSPC quirk worked around by plink-stdin streaming.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds quality-filter controls to the search UI: a "min score" select
(any/2+/3+/4+/5) and a "hide banter" checkbox. Q/A hits gain a small
color-coded usefulness badge (1-5, red->green) and a topic_class tag
(computer-help, banter, off-topic, promo). Low-score and banter rows
render dimmed by default so they're visible but de-emphasized.
Defaults to "any" + banter visible to preserve existing search habits.
Mike toggles up when he wants quality. URL-encoded params built via
URLSearchParams so empty values don't leak into requests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Streams the read-only archive.db over the same Tailscale-routed port
as the search service. Companion to azcomputerguru/radio-archive-portable
which curl-fetches from this endpoint and runs locally on the laptop.
Disclosure equivalent to /api/search (which already exposes every
transcript), so no auth added. Deployed to Jupiter; verified GET
returns 60 MB SQLite blob with all 1,405 classifier rows intact.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3.5h run on qwen3:14b processed 1,405/1,407 Q/A pairs (2 failed,
will retry on next invocation). 37% scored 4-5 (useful), 41%
scored 1-2 (banter/promo/off-topic). API filter ready; Jupiter
redeploy pending Mike's manual review.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an Ollama-based content quality classifier and exposes the
results via the search API. 1,407 existing Q/A pairs were scored
in 3.5h via qwen3:14b (1,405 succeeded, 2 failed).
Distribution: 37% scored 4-5 (useful), 41% scored 1-2 (banter/promo/
off-topic). 43% flagged as banter overall. Default-on filtering at
search time will hide ~half of the noise without losing any real
listener questions.
Files:
- new classify_qa_quality.py: walks qa_pairs, calls Ollama qwen3:14b
per row, writes usefulness_score/topic_class/is_banter back to DB.
Idempotent (--rebuild to reprocess), --smoke for sample check, --limit
for partial runs. Detached run handles 1407 rows in ~3.5h on a 4090.
- server/main.py: /api/search accepts min_score (0-5) and exclude_banter
query params. NULL scores treat as "include" so unprocessed rows still
appear. Episode detail endpoint includes the new fields in qa results.
Schema migration in import_to_sqlite.py was made by the same agent run
(visible on the live archive.db: usefulness_score / topic_class /
is_banter columns now exist on qa_pairs).
Local archive.db updated; Jupiter container has NOT been redeployed
yet — that is a separate manual step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
role_assigned() only checks direct/permanent roleAssignments.
PIM-managed assignments are in roleAssignmentSchedules and won't
be found, producing noisy (non-blocking) output on re-runs against
tenants with PIM-assigned roles (e.g. Cascades).
TODO comment added at the helper — Howard to implement the fix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Six small bash scripts uploaded to /tmp on 172.16.3.22 during the
OwnCloud cron stacking incident — investigation, group enumeration,
failed group-restrict attempt, occ subcommand discovery. Captured for
audit; full context in clients/pavon/session-logs/2026-04-29-session.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Found 75-126 stale `occ system:cron` processes on 172.16.3.22 piling up
since 2026-04-27 due to bad oc_filecache LIKE query against pavon's 257K
camera files. Killed stale procs (load 80 -> 5), wrapped apache crontab
with `flock -n /tmp/oc-cron.lock` to prevent restacking. Per-user
versioning disable rejected by OwnCloud Community (`files_versions`
can't be enabled for groups); workaround `occ versions:cleanup pavon`
identified and deferred. Migration/retention cron deferred per user.
NVR architecture clarified: GeoVision NVRs sync via OC Desktop client
with virtual file placeholders; no direct SMB access to Jupiter.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Day-long session unblocking the Cascades CA reconciliation that was paused on
the Tenant Admin SP directory-role gap. Discovered Microsoft also tightened
the OAuth scope for /identity/conditionalAccess/* reads (Policy.Read.All now
required, Policy.ReadWrite.ConditionalAccess no longer accepted for reads).
Patched Tenant Admin manifest accordingly and re-consented in Cascades.
Phase B Intune state turned out to be far more built than the 4/20 log
suggested -- compliance policy, Wi-Fi, device restrictions, both SDM app
configs (Authenticator + Teams), and 7 of 8 apps were already deployed and
assigned. PATCHed device restrictions to block camera/Bluetooth/roaming
and enabled Managed Home Screen multi-app kiosk (ALIS + Teams visible,
10-min auto-signout). PATCHed Cascades named location to add primary WAN
(184.191.143.62/32). Howard added Outlook from Managed Play; SMB encryption
enabled on \CS-SERVER\homes.
CA bypass design corrected -- original §5 plan in user-account-rollout-plan.md
called for "block off-site + MFA on-site" which doesn't match the actual goal
of bypass when network + device assurance present. Reshaped to three policies
that produce on-site-compliant = password only, anything else = MFA or block.
onboard-tenant.sh patched to:
1. Backfill Policy.Read.All on Tenant Admin SP if missing (idempotent --
for tenants consented before the 2026-04-29 manifest update).
2. Assign Conditional Access Administrator directory role to Tenant Admin
SP at onboard time. Mirrors the Exchange Operator fix Mike landed in
16f95e8.
Validated with --dry-run against Cascades. Customer-facing tenants already
onboarded should be re-run with this script to backfill both items.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Added cross-reference from FEATURE_ROADMAP.md to UI_GAPS.md tracking document.
Clarifies that features may be backend-complete but UI-incomplete.
Submodule commit: f76051a
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Documented two fundamental GuruRMM development principles:
1. Holistic Feature Development (MANDATORY):
- Every feature requires complete stack: backend, API, UI/UX, docs
- Features without management interfaces are incomplete
- Design for scalability and future expansion
- Example workflows included
2. AI-Optional Operation:
- Product must work without AI agents (Claude, autonomous tools)
- AI features are enhancements, not requirements
- Core operations remain deterministic and reliable
Principles documented in guru-rmm/docs/DESIGN.md and now in memory for
cross-session reference.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Updated DESIGN.md with two fundamental principles:
1. Holistic Feature Development - every feature needs full stack (backend, API, UI, docs)
2. AI-Optional Operation - product works without AI agents; AI features are enhancements
Submodule commit: e490307
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Updated GuruRMM roadmap with two major features:
- Network Discovery Node (P2): site-level device discovery and mapping
- Local Collection Node (P2): reduce WAN traffic by local aggregation
Submodule commit: db7d074
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Howard is cleared to proceed with Path A (Graph API role assignment) for
Cascades CA Administrator fix.
Also communicated new approval workflow:
- General tools: Howard can modify OR Claude can execute with Howard/Mike approval
- Projects: require Mike approval, features→roadmap, bugs→bug list
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Tools (remediation-tool, onboard scripts, MSP utilities):
- Howard can modify directly
- Claude can execute with Howard OR Mike approval
- No roadmap process, immediate operational changes
Projects (GuruRMM, ClaudeTools API, etc.):
- Require Mike approval
- Features go to roadmap
- Bugs go to bug list
Established during Cascades CA role gap fix discussion.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
