claudetools/GREPAI_SYNC_STRATEGY.md

# Grepai Sync Strategy

**Purpose:** Keep grepai indexes synchronized between Windows and Mac development machines

---

## Understanding Grepai Index

**What is the index?**
- Semantic embeddings of your codebase (13,020 chunks from 961 files)
- Size: 73.7 MB
- Generated using: nomic-embed-text model via Ollama
- Stored locally: `.grepai/` directory (usually)

**Index components:**
- Embeddings database (vector representations of code)
- Symbol tracking database (functions, classes, etc.)
- File metadata (paths, timestamps, hashes)

---

## Sync Strategy Options

### Option 1: Independent Indexes (RECOMMENDED)

**How it works:**
- Each machine maintains its own grepai index
- Index is gitignored (not committed to repository)
- Each machine rebuilds index from local codebase

**Advantages:**
- [OK] Always consistent with local codebase
- [OK] No merge conflicts
- [OK] Handles machine-specific paths correctly
- [OK] Simple and reliable

**Disadvantages:**
- [WARNING] Must rebuild index on each machine (one-time setup)
- [WARNING] Initial indexing takes time (~2-5 minutes for 961 files)

**Setup:**

```bash
# Add to .gitignore
echo ".grepai/" >> .gitignore

# On each machine:
grepai init
grepai index

# Keep codebase in sync via git
git pull origin main
grepai index  # Rebuild after pulling changes
```

**When to rebuild:**
- After pulling major code changes (>50 files)
- After switching branches
- If search results seem outdated
- Weekly maintenance (optional)

---

### Option 2: Shared Index via Git

**How it works:**
- Commit `.grepai/` directory to repository
- Pull index along with code changes

**Advantages:**
- [OK] Instant sync (no rebuild needed)
- [OK] Same index on all machines

**Disadvantages:**
- [ERROR] Can cause merge conflicts
- [ERROR] May have absolute path issues (D:\ vs ~/)
- [ERROR] Index may get out of sync with actual code
- [ERROR] Increases repository size (+73.7 MB)

**NOT RECOMMENDED** due to path conflicts and sync issues.

---

### Option 3: Automated Rebuild on Pull (BEST PRACTICE)

**How it works:**
- Keep indexes independent (Option 1)
- Automatically rebuild index after git pull
- Use git hooks to trigger rebuild

**Setup:**

Create `.git/hooks/post-merge` (git pull trigger):

```bash
#!/bin/bash
echo "[grepai] Rebuilding index after merge..."
grepai index --quiet
echo "[OK] Index updated"
```

Make executable:
```bash
chmod +x .git/hooks/post-merge
```

**Advantages:**
- [OK] Always up to date
- [OK] Automated (no manual intervention)
- [OK] No merge conflicts
- [OK] Each machine has correct index

**Disadvantages:**
- [WARNING] Adds 1-2 minutes to git pull time
- [WARNING] Requires git hook setup on each machine

---

## Recommended Workflow

### Initial Setup (One-Time Per Machine)

**On Windows:**
```bash
# Ensure .grepai is gitignored
echo ".grepai/" >> .gitignore
git add .gitignore
git commit -m "chore: gitignore grepai index"

# Build index
grepai index
```

**On Mac:**
```bash
# Pull latest code
git pull origin main

# Install Ollama models
ollama pull nomic-embed-text

# Build index
grepai index
```

### Daily Workflow

**Start of day (on either machine):**
```bash
# Update codebase
git pull origin main

# Rebuild index (if significant changes)
grepai index
```

**During development:**
- No action needed
- Grepai auto-updates as you edit files (depending on configuration)

**End of day:**
```bash
# Commit your changes
git add .
git commit -m "your message"
git push origin main
```

**On other machine:**
```bash
# Pull changes
git pull origin main

# Rebuild index
grepai index
```

---

## Quick Rebuild Commands

**Full rebuild:**
```bash
grepai index
```

**Incremental update (faster, if supported):**
```bash
grepai index --incremental
```

**Check if rebuild needed:**
```bash
# Compare last index time with last git pull
grepai status
git log -1 --format="%ai"
```

---

## Automation Script

**Create `sync-and-index.sh`:**

```bash
#!/bin/bash
# Sync codebase and rebuild grepai index

echo "=== Syncing ClaudeTools ==="

# Pull latest changes
echo "[1/3] Pulling from git..."
git pull origin main

if [ $? -ne 0 ]; then
    echo "[ERROR] Git pull failed"
    exit 1
fi

# Check if significant changes
CHANGED_FILES=$(git diff HEAD@{1} --name-only | wc -l)
echo "[2/3] Changed files: $CHANGED_FILES"

# Rebuild index if changes detected
if [ "$CHANGED_FILES" -gt 0 ]; then
    echo "[3/3] Rebuilding grepai index..."
    grepai index
    echo "[OK] Sync complete with index rebuild"
else
    echo "[3/3] No changes, skipping index rebuild"
    echo "[OK] Sync complete"
fi
```

**Usage:**
```bash
chmod +x sync-and-index.sh
./sync-and-index.sh
```

---

## Monitoring Index Health

**Check index status:**
```bash
grepai status
```

**Expected output (healthy):**
```
Total files: 961
Total chunks: 13,020
Index size: 73.7 MB
Last updated: [recent timestamp]
Provider: ollama
Model: nomic-embed-text
Symbols: Ready
```

**Signs of unhealthy index:**
- File count doesn't match codebase
- Last updated > 7 days old
- Symbol tracking not ready
- Search results seem wrong

**Fix:**
```bash
grepai rebuild  # or
grepai index --force
```

---

## Best Practices

1. **Always gitignore `.grepai/`** - Prevents merge conflicts
2. **Rebuild after major pulls** - Keeps index accurate
3. **Use same embedding model** - Ensures consistency (nomic-embed-text)
4. **Verify index health weekly** - Run `grepai status`
5. **Document rebuild frequency** - Set team expectations

---

## Troubleshooting

### Index out of sync
```bash
# Force complete rebuild
rm -rf .grepai
grepai init
grepai index
```

### Different results on different machines
- Check embedding model: `grepai status | grep model`
- Should both use: `nomic-embed-text`
- Rebuild with same model if different

### Index too large
```bash
# Check what's being indexed
grepai stats

# Add exclusions to .grepai.yml (if exists)
# exclude:
#   - node_modules/
#   - venv/
#   - .git/
```

---

## Summary

**RECOMMENDED APPROACH: Option 3 (Automated Rebuild)**

**Setup:**
1. Gitignore `.grepai/` directory
2. Install git hook for post-merge rebuild
3. Each machine maintains independent index
4. Index rebuilds automatically after git pull

**Maintenance:**
- Initial index build: 2-5 minutes (one-time per machine)
- Incremental rebuilds: 30-60 seconds (after pulls)
- Full rebuilds: As needed (weekly or when issues arise)

**Key principle:** Treat grepai index like compiled artifacts - gitignore them and rebuild from source (the codebase) as needed.

---

## Last Updated

2026-01-22 - Initial creation