Remove conversation context/recall system from ClaudeTools
Completely removed the database context recall system while preserving database tables for safety. This major cleanup removes 80+ files and 16,831 lines of code. What was removed: - API layer: 4 routers (conversation-contexts, context-snippets, project-states, decision-logs) with 35+ endpoints - Database models: 5 models (ConversationContext, ContextSnippet, DecisionLog, ProjectState, ContextTag) - Services: 4 service layers with business logic - Schemas: 4 Pydantic schema files - Claude Code hooks: 13 hook files (user-prompt-submit, task-complete, sync-contexts, periodic saves) - Scripts: 15+ scripts (import, migration, testing, tombstone checking) - Tests: 5 test files (context recall, compression, diagnostics) - Documentation: 30+ markdown files (guides, architecture, quick starts) - Utilities: context compression, conversation parsing Files modified: - api/main.py: Removed router registrations - api/models/__init__.py: Removed model imports - api/schemas/__init__.py: Removed schema imports - api/services/__init__.py: Removed service imports - .claude/claude.md: Completely rewritten without context references Database tables preserved: - conversation_contexts, context_snippets, context_tags, project_states, decision_logs (5 orphaned tables remain for safety) - Migration created but NOT applied: 20260118_172743_remove_context_system.py - Tables can be dropped later when confirmed not needed New files added: - CONTEXT_SYSTEM_REMOVAL_SUMMARY.md: Detailed removal report - CONTEXT_SYSTEM_REMOVAL_COMPLETE.md: Final status - CONTEXT_EXPORT_RESULTS.md: Export attempt results - scripts/export-tombstoned-contexts.py: Export tool for future use - migrations/versions/20260118_172743_remove_context_system.py Impact: - Reduced from 130 to 95 API endpoints - Reduced from 43 to 38 active database tables - Removed 16,831 lines of code - System fully operational without context recall Reason for removal: - System was not actively used (no tombstoned contexts found) - Reduces codebase complexity - Focuses on core MSP work tracking functionality - Database preserved for safety (can rollback if needed) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
283
.claude/DATABASE_FIRST_PROTOCOL.md
Normal file
283
.claude/DATABASE_FIRST_PROTOCOL.md
Normal file
@@ -0,0 +1,283 @@
|
||||
# Database-First Protocol
|
||||
|
||||
**CRITICAL:** This protocol MUST be followed for EVERY user request.
|
||||
|
||||
---
|
||||
|
||||
## The Problem
|
||||
|
||||
Currently, Claude:
|
||||
1. Receives user request
|
||||
2. Searches local files (maybe)
|
||||
3. Performs work
|
||||
4. (Never saves context automatically)
|
||||
|
||||
This wastes tokens, misses critical context, and loses work across sessions.
|
||||
|
||||
---
|
||||
|
||||
## The Solution: Database-First Protocol
|
||||
|
||||
### MANDATORY FIRST STEP - For EVERY User Request
|
||||
|
||||
```
|
||||
BEFORE doing ANYTHING else:
|
||||
|
||||
1. Query the context database for relevant information
|
||||
2. Inject retrieved context into your working memory
|
||||
3. THEN proceed with the user's request
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation
|
||||
|
||||
### Step 1: Check Database (ALWAYS FIRST)
|
||||
|
||||
Before analyzing the user's request, run this command:
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer $JWT_TOKEN" \
|
||||
"http://172.16.3.30:8001/api/conversation-contexts/recall?\
|
||||
search_term={user_keywords}&limit=10" | python -m json.tool
|
||||
```
|
||||
|
||||
Extract keywords from user request. Examples:
|
||||
- User: "What's the status of Dataforth project?" → search_term=dataforth
|
||||
- User: "Continue work on GuruConnect" → search_term=guruconnect
|
||||
- User: "Fix the API bug" → search_term=API+bug
|
||||
- User: "Help with database" → search_term=database
|
||||
|
||||
### Step 2: Review Retrieved Context
|
||||
|
||||
The API returns up to 10 relevant contexts with:
|
||||
- `title` - Short description
|
||||
- `dense_summary` - Compressed context (90% token reduction)
|
||||
- `relevance_score` - How relevant (0-10)
|
||||
- `tags` - Keywords for filtering
|
||||
- `created_at` - Timestamp
|
||||
|
||||
### Step 3: Use Context in Your Response
|
||||
|
||||
Reference the context when responding:
|
||||
- "Based on previous context from {date}..."
|
||||
- "According to the database, Dataforth DOS project..."
|
||||
- "Context shows this was last discussed on..."
|
||||
|
||||
### Step 4: Save New Context (After Completion)
|
||||
|
||||
After completing a significant task:
|
||||
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer $JWT_TOKEN" \
|
||||
-X POST "http://172.16.3.30:8001/api/conversation-contexts" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"project_id": "c3d9f1c8-dc2b-499f-a228-3a53fa950e7b",
|
||||
"context_type": "session_summary",
|
||||
"title": "Brief title of what was accomplished",
|
||||
"dense_summary": "Compressed summary of work done, decisions made, files changed",
|
||||
"relevance_score": 7.0,
|
||||
"tags": "[\"keyword1\", \"keyword2\", \"keyword3\"]"
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## When to Save Context
|
||||
|
||||
Save context automatically when:
|
||||
|
||||
1. **Task Completion** - TodoWrite task marked as completed
|
||||
2. **Major Decision** - Architectural choice, approach selection
|
||||
3. **File Changes** - Significant code changes (>50 lines)
|
||||
4. **Problem Solved** - Bug fixed, issue resolved
|
||||
5. **User Requests** - Via /snapshot command
|
||||
6. **Session End** - Before closing conversation
|
||||
|
||||
---
|
||||
|
||||
## Agent Delegation Rules
|
||||
|
||||
**Main Claude is a COORDINATOR, not an EXECUTOR.**
|
||||
|
||||
Before performing any task, check delegation table:
|
||||
|
||||
| Task Type | Delegate To | Always? |
|
||||
|-----------|-------------|---------|
|
||||
| Context retrieval | Database Agent | ✅ YES |
|
||||
| Codebase search | Explore Agent | For patterns/keywords |
|
||||
| Code changes >10 lines | Coding Agent | ✅ YES |
|
||||
| Running tests | Testing Agent | ✅ YES |
|
||||
| Git operations | Gitea Agent | ✅ YES |
|
||||
| File operations <5 files | Main Claude | Direct OK |
|
||||
| Documentation | Documentation Squire | For comprehensive docs |
|
||||
|
||||
**How to Delegate:**
|
||||
|
||||
```
|
||||
Instead of: Searching files directly with Grep/Glob
|
||||
Do: "Let me delegate to the Explore agent to search the codebase..."
|
||||
|
||||
Instead of: Writing code directly
|
||||
Do: "Let me delegate to the Coding Agent to implement this change..."
|
||||
|
||||
Instead of: Running tests yourself
|
||||
Do: "Let me delegate to the Testing Agent to run the test suite..."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Context Database Quick Reference
|
||||
|
||||
### Query Endpoints
|
||||
|
||||
```bash
|
||||
# Search by term
|
||||
GET /api/conversation-contexts/recall?search_term={term}&limit=10
|
||||
|
||||
# Filter by tags
|
||||
GET /api/conversation-contexts/recall?tags=dataforth&tags=dos&limit=10
|
||||
|
||||
# Get by project
|
||||
GET /api/conversation-contexts/recall?project_id={uuid}&limit=10
|
||||
|
||||
# List all recent
|
||||
GET /api/conversation-contexts?limit=50
|
||||
```
|
||||
|
||||
### Save Endpoint
|
||||
|
||||
```bash
|
||||
POST /api/conversation-contexts
|
||||
{
|
||||
"project_id": "uuid",
|
||||
"context_type": "session_summary|checkpoint|decision|problem_solution",
|
||||
"title": "Short title",
|
||||
"dense_summary": "Compressed summary with key info",
|
||||
"relevance_score": 1.0-10.0,
|
||||
"tags": "[\"tag1\", \"tag2\"]"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example Workflow
|
||||
|
||||
### User Request: "What's the status of the Dataforth DOS project?"
|
||||
|
||||
**WRONG Approach:**
|
||||
```
|
||||
Claude: Let me search local files...
|
||||
(Wastes tokens, misses imported context in database)
|
||||
```
|
||||
|
||||
**CORRECT Approach:**
|
||||
```
|
||||
Claude: Let me check the context database first...
|
||||
|
||||
[Runs: curl .../recall?search_term=dataforth]
|
||||
|
||||
Claude: "Based on context retrieved from the database, the Dataforth
|
||||
DOS machines project involves analyzing drive images from test machines
|
||||
with ATE (Automated Test Equipment) software. The conversation was
|
||||
imported on 2026-01-18 and includes 1,241KB of data.
|
||||
|
||||
The project appears to focus on Dataforth industrial I/O equipment
|
||||
testing (5B, 7B, 8B series modules).
|
||||
|
||||
Would you like me to delegate to the Explore agent to find specific
|
||||
files related to this project?"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integration with Hooks
|
||||
|
||||
The hooks in `.claude/hooks/` should assist but NOT replace manual queries:
|
||||
|
||||
- `user-prompt-submit` - Auto-injects context (passive)
|
||||
- `task-complete` - Auto-saves context (passive)
|
||||
|
||||
**BUT:** You should ACTIVELY query database yourself before major work.
|
||||
|
||||
Don't rely solely on hooks. They're a backup, not the primary mechanism.
|
||||
|
||||
---
|
||||
|
||||
## Token Efficiency
|
||||
|
||||
### Before Database-First:
|
||||
- Read 3MB of local files: ~750,000 tokens
|
||||
- Parse conversation histories: ~250,000 tokens
|
||||
- **Total:** ~1,000,000 tokens per session
|
||||
|
||||
### After Database-First:
|
||||
- Query database: 500 tokens (API call)
|
||||
- Receive compressed summaries: ~5,000 tokens (10 contexts)
|
||||
- **Total:** ~5,500 tokens per session
|
||||
|
||||
**Savings:** 99.4% token reduction
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Database Query Returns Empty
|
||||
|
||||
```bash
|
||||
# Check if API is up
|
||||
curl http://172.16.3.30:8001/health
|
||||
|
||||
# Check total contexts
|
||||
curl -H "Authorization: Bearer $JWT" \
|
||||
http://172.16.3.30:8001/api/conversation-contexts | \
|
||||
python -c "import sys,json; print(f'Total: {json.load(sys.stdin)[\"total\"]}')"
|
||||
|
||||
# Try different search term
|
||||
# Instead of: search_term=dataforth%20DOS
|
||||
# Try: search_term=dataforth
|
||||
```
|
||||
|
||||
### Authentication Fails
|
||||
|
||||
```bash
|
||||
# Check JWT token in config
|
||||
cat .claude/context-recall-config.env | grep JWT_TOKEN
|
||||
|
||||
# Verify token not expired
|
||||
# Current token expires: 2026-02-16
|
||||
```
|
||||
|
||||
### No Results for Known Project
|
||||
|
||||
The recall endpoint uses PostgreSQL full-text search. Try:
|
||||
- Simpler search terms
|
||||
- Individual keywords instead of phrases
|
||||
- Checking tags directly: `?tags=dataforth`
|
||||
|
||||
---
|
||||
|
||||
## Enforcement
|
||||
|
||||
This protocol is MANDATORY. To ensure compliance:
|
||||
|
||||
1. **Every response** should start with "Checking database for context..."
|
||||
2. **Before major work**, always query database
|
||||
3. **After completion**, always save summary
|
||||
4. **For delegation**, use agents not direct execution
|
||||
|
||||
**Violation Example:**
|
||||
```
|
||||
User: "Find all Python files"
|
||||
Claude: [Runs Glob directly] ❌ WRONG
|
||||
|
||||
Correct:
|
||||
Claude: "Let me delegate to Explore agent to search for Python files" ✅
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-01-18
|
||||
**Status:** ACTIVE - MUST BE FOLLOWED
|
||||
**Priority:** CRITICAL
|
||||
Reference in New Issue
Block a user