claudetools/.claude/DATABASE_FIRST_PROTOCOL.md

# Database-First Protocol

**CRITICAL:** This protocol MUST be followed for EVERY user request.

---

## The Problem

Currently, Claude:
1. Receives user request
2. Searches local files (maybe)
3. Performs work
4. (Never saves context automatically)

This wastes tokens, misses critical context, and loses work across sessions.

---

## The Solution: Database-First Protocol

### MANDATORY FIRST STEP - For EVERY User Request

```
BEFORE doing ANYTHING else:

1. Query the context database for relevant information
2. Inject retrieved context into your working memory
3. THEN proceed with the user's request
```

---

## Implementation

### Step 1: Check Database (ALWAYS FIRST)

Before analyzing the user's request, run this command:

```bash
curl -s -H "Authorization: Bearer $JWT_TOKEN" \
  "http://172.16.3.30:8001/api/conversation-contexts/recall?\
search_term={user_keywords}&limit=10" | python -m json.tool
```

Extract keywords from user request. Examples:
- User: "What's the status of Dataforth project?" → search_term=dataforth
- User: "Continue work on GuruConnect" → search_term=guruconnect
- User: "Fix the API bug" → search_term=API+bug
- User: "Help with database" → search_term=database

### Step 2: Review Retrieved Context

The API returns up to 10 relevant contexts with:
- `title` - Short description
- `dense_summary` - Compressed context (90% token reduction)
- `relevance_score` - How relevant (0-10)
- `tags` - Keywords for filtering
- `created_at` - Timestamp

### Step 3: Use Context in Your Response

Reference the context when responding:
- "Based on previous context from {date}..."
- "According to the database, Dataforth DOS project..."
- "Context shows this was last discussed on..."

### Step 4: Save New Context (After Completion)

After completing a significant task:

```bash
curl -s -H "Authorization: Bearer $JWT_TOKEN" \
  -X POST "http://172.16.3.30:8001/api/conversation-contexts" \
  -H "Content-Type: application/json" \
  -d '{
    "project_id": "c3d9f1c8-dc2b-499f-a228-3a53fa950e7b",
    "context_type": "session_summary",
    "title": "Brief title of what was accomplished",
    "dense_summary": "Compressed summary of work done, decisions made, files changed",
    "relevance_score": 7.0,
    "tags": "[\"keyword1\", \"keyword2\", \"keyword3\"]"
  }'
```

---

## When to Save Context

Save context automatically when:

1. **Task Completion** - TodoWrite task marked as completed
2. **Major Decision** - Architectural choice, approach selection
3. **File Changes** - Significant code changes (>50 lines)
4. **Problem Solved** - Bug fixed, issue resolved
5. **User Requests** - Via /snapshot command
6. **Session End** - Before closing conversation

---

## Agent Delegation Rules

**Main Claude is a COORDINATOR, not an EXECUTOR.**

Before performing any task, check delegation table:

| Task Type | Delegate To | Always? |
|-----------|-------------|---------|
| Context retrieval | Database Agent | [OK] YES |
| Codebase search | Explore Agent | For patterns/keywords |
| Code changes >10 lines | Coding Agent | [OK] YES |
| Running tests | Testing Agent | [OK] YES |
| Git operations | Gitea Agent | [OK] YES |
| File operations <5 files | Main Claude | Direct OK |
| Documentation | Documentation Squire | For comprehensive docs |

**How to Delegate:**

```
Instead of: Searching files directly with Grep/Glob
Do: "Let me delegate to the Explore agent to search the codebase..."

Instead of: Writing code directly
Do: "Let me delegate to the Coding Agent to implement this change..."

Instead of: Running tests yourself
Do: "Let me delegate to the Testing Agent to run the test suite..."
```

---

## Context Database Quick Reference

### Query Endpoints

```bash
# Search by term
GET /api/conversation-contexts/recall?search_term={term}&limit=10

# Filter by tags
GET /api/conversation-contexts/recall?tags=dataforth&tags=dos&limit=10

# Get by project
GET /api/conversation-contexts/recall?project_id={uuid}&limit=10

# List all recent
GET /api/conversation-contexts?limit=50
```

### Save Endpoint

```bash
POST /api/conversation-contexts
{
  "project_id": "uuid",
  "context_type": "session_summary|checkpoint|decision|problem_solution",
  "title": "Short title",
  "dense_summary": "Compressed summary with key info",
  "relevance_score": 1.0-10.0,
  "tags": "[\"tag1\", \"tag2\"]"
}
```

---

## Example Workflow

### User Request: "What's the status of the Dataforth DOS project?"

**WRONG Approach:**
```
Claude: Let me search local files...
(Wastes tokens, misses imported context in database)
```

**CORRECT Approach:**
```
Claude: Let me check the context database first...

[Runs: curl .../recall?search_term=dataforth]

Claude: "Based on context retrieved from the database, the Dataforth
DOS machines project involves analyzing drive images from test machines
with ATE (Automated Test Equipment) software. The conversation was
imported on 2026-01-18 and includes 1,241KB of data.

The project appears to focus on Dataforth industrial I/O equipment
testing (5B, 7B, 8B series modules).

Would you like me to delegate to the Explore agent to find specific
files related to this project?"
```

---

## Integration with Hooks

The hooks in `.claude/hooks/` should assist but NOT replace manual queries:

- `user-prompt-submit` - Auto-injects context (passive)
- `task-complete` - Auto-saves context (passive)

**BUT:** You should ACTIVELY query database yourself before major work.

Don't rely solely on hooks. They're a backup, not the primary mechanism.

---

## Token Efficiency

### Before Database-First:
- Read 3MB of local files: ~750,000 tokens
- Parse conversation histories: ~250,000 tokens
- **Total:** ~1,000,000 tokens per session

### After Database-First:
- Query database: 500 tokens (API call)
- Receive compressed summaries: ~5,000 tokens (10 contexts)
- **Total:** ~5,500 tokens per session

**Savings:** 99.4% token reduction

---

## Troubleshooting

### Database Query Returns Empty

```bash
# Check if API is up
curl http://172.16.3.30:8001/health

# Check total contexts
curl -H "Authorization: Bearer $JWT" \
  http://172.16.3.30:8001/api/conversation-contexts | \
  python -c "import sys,json; print(f'Total: {json.load(sys.stdin)[\"total\"]}')"

# Try different search term
# Instead of: search_term=dataforth%20DOS
# Try: search_term=dataforth
```

### Authentication Fails

```bash
# Check JWT token in config
cat .claude/context-recall-config.env | grep JWT_TOKEN

# Verify token not expired
# Current token expires: 2026-02-16
```

### No Results for Known Project

The recall endpoint uses PostgreSQL full-text search. Try:
- Simpler search terms
- Individual keywords instead of phrases
- Checking tags directly: `?tags=dataforth`

---

## Enforcement

This protocol is MANDATORY. To ensure compliance:

1. **Every response** should start with "Checking database for context..."
2. **Before major work**, always query database
3. **After completion**, always save summary
4. **For delegation**, use agents not direct execution

**Violation Example:**
```
User: "Find all Python files"
Claude: [Runs Glob directly] [ERROR] WRONG

Correct:
Claude: "Let me delegate to Explore agent to search for Python files" [OK]
```

---

**Last Updated:** 2026-01-18
**Status:** ACTIVE - MUST BE FOLLOWED
**Priority:** CRITICAL