Remove conversation context/recall system from ClaudeTools
Completely removed the database context recall system while preserving database tables for safety. This major cleanup removes 80+ files and 16,831 lines of code. What was removed: - API layer: 4 routers (conversation-contexts, context-snippets, project-states, decision-logs) with 35+ endpoints - Database models: 5 models (ConversationContext, ContextSnippet, DecisionLog, ProjectState, ContextTag) - Services: 4 service layers with business logic - Schemas: 4 Pydantic schema files - Claude Code hooks: 13 hook files (user-prompt-submit, task-complete, sync-contexts, periodic saves) - Scripts: 15+ scripts (import, migration, testing, tombstone checking) - Tests: 5 test files (context recall, compression, diagnostics) - Documentation: 30+ markdown files (guides, architecture, quick starts) - Utilities: context compression, conversation parsing Files modified: - api/main.py: Removed router registrations - api/models/__init__.py: Removed model imports - api/schemas/__init__.py: Removed schema imports - api/services/__init__.py: Removed service imports - .claude/claude.md: Completely rewritten without context references Database tables preserved: - conversation_contexts, context_snippets, context_tags, project_states, decision_logs (5 orphaned tables remain for safety) - Migration created but NOT applied: 20260118_172743_remove_context_system.py - Tables can be dropped later when confirmed not needed New files added: - CONTEXT_SYSTEM_REMOVAL_SUMMARY.md: Detailed removal report - CONTEXT_SYSTEM_REMOVAL_COMPLETE.md: Final status - CONTEXT_EXPORT_RESULTS.md: Export attempt results - scripts/export-tombstoned-contexts.py: Export tool for future use - migrations/versions/20260118_172743_remove_context_system.py Impact: - Reduced from 130 to 95 API endpoints - Reduced from 43 to 38 active database tables - Removed 16,831 lines of code - System fully operational without context recall Reason for removal: - System was not actively used (no tombstoned contexts found) - Reduces codebase complexity - Focuses on core MSP work tracking functionality - Database preserved for safety (can rollback if needed) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
214
scripts/README.md
Normal file
214
scripts/README.md
Normal file
@@ -0,0 +1,214 @@
|
||||
# ClaudeTools Scripts
|
||||
|
||||
Utility scripts for managing the ClaudeTools system.
|
||||
|
||||
---
|
||||
|
||||
## Core Scripts
|
||||
|
||||
### Context Recall System
|
||||
|
||||
**`setup-context-recall.sh`**
|
||||
- One-time setup for context recall system
|
||||
- Configures JWT authentication
|
||||
- Tests API connectivity
|
||||
|
||||
**`test-context-recall.sh`**
|
||||
- Verify context recall system functionality
|
||||
- Test API endpoints
|
||||
- Check compression
|
||||
|
||||
### Conversation Import & Archive
|
||||
|
||||
**`import-conversations.py`**
|
||||
- Import conversation JSONL files to database
|
||||
- Extract and compress conversation context
|
||||
- Tag extraction and categorization
|
||||
- Optional tombstone creation with `--create-tombstones`
|
||||
|
||||
**`archive-imported-conversations.py`**
|
||||
- Archive imported conversation files
|
||||
- Create tombstone markers
|
||||
- Move files to archived/ subdirectories
|
||||
- Database verification (optional)
|
||||
|
||||
**`check-tombstones.py`**
|
||||
- Verify tombstone integrity
|
||||
- Validate JSON structure
|
||||
- Check archived files exist
|
||||
- Database context verification
|
||||
|
||||
**`TOMBSTONE_QUICK_START.md`**
|
||||
- Quick reference for tombstone system
|
||||
- Common commands
|
||||
- Troubleshooting tips
|
||||
|
||||
### Database & Testing
|
||||
|
||||
**`test_db_connection.py`**
|
||||
- Test database connectivity
|
||||
- Verify credentials
|
||||
- Check table access
|
||||
|
||||
**`test-server.sh`**
|
||||
- Start development server
|
||||
- Run basic API tests
|
||||
|
||||
### MCP Servers
|
||||
|
||||
**`setup-mcp-servers.sh`**
|
||||
- Configure Model Context Protocol servers
|
||||
- Setup GitHub, Filesystem, Sequential Thinking
|
||||
|
||||
---
|
||||
|
||||
## Tombstone System (NEW)
|
||||
|
||||
Archive imported conversation files with small marker files.
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
# Archive all imported files
|
||||
python scripts/archive-imported-conversations.py --skip-verification
|
||||
|
||||
# Verify tombstones
|
||||
python scripts/check-tombstones.py
|
||||
|
||||
# Check space savings
|
||||
du -sh imported-conversations/
|
||||
```
|
||||
|
||||
### Documentation
|
||||
|
||||
- `TOMBSTONE_QUICK_START.md` - Quick reference
|
||||
- `../TOMBSTONE_SYSTEM.md` - Complete documentation
|
||||
- `../TOMBSTONE_IMPLEMENTATION_SUMMARY.md` - Implementation details
|
||||
|
||||
### Expected Results
|
||||
|
||||
- 549 tombstone files (~1 MB)
|
||||
- 549 archived files in subdirectories
|
||||
- 99%+ space reduction in active directory
|
||||
|
||||
---
|
||||
|
||||
## Usage Patterns
|
||||
|
||||
### Initial Setup
|
||||
|
||||
```bash
|
||||
# 1. Setup context recall
|
||||
bash scripts/setup-context-recall.sh
|
||||
|
||||
# 2. Setup MCP servers (optional)
|
||||
bash scripts/setup-mcp-servers.sh
|
||||
|
||||
# 3. Test database connection
|
||||
python scripts/test_db_connection.py
|
||||
```
|
||||
|
||||
### Import Conversations
|
||||
|
||||
```bash
|
||||
# Import with automatic tombstones
|
||||
python scripts/import-conversations.py --create-tombstones
|
||||
|
||||
# Or import first, archive later
|
||||
python scripts/import-conversations.py
|
||||
python scripts/archive-imported-conversations.py --skip-verification
|
||||
```
|
||||
|
||||
### Verify System
|
||||
|
||||
```bash
|
||||
# Test context recall
|
||||
bash scripts/test-context-recall.sh
|
||||
|
||||
# Check tombstones
|
||||
python scripts/check-tombstones.py
|
||||
|
||||
# Test API
|
||||
bash scripts/test-server.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Script Categories
|
||||
|
||||
### Setup & Configuration
|
||||
- `setup-context-recall.sh`
|
||||
- `setup-mcp-servers.sh`
|
||||
|
||||
### Import & Archive
|
||||
- `import-conversations.py`
|
||||
- `archive-imported-conversations.py`
|
||||
- `check-tombstones.py`
|
||||
|
||||
### Testing & Verification
|
||||
- `test_db_connection.py`
|
||||
- `test-context-recall.sh`
|
||||
- `test-server.sh`
|
||||
- `test-tombstone-system.sh`
|
||||
|
||||
### Utilities
|
||||
- Various helper scripts
|
||||
|
||||
---
|
||||
|
||||
## Common Commands
|
||||
|
||||
```bash
|
||||
# Start API server
|
||||
uvicorn api.main:app --reload
|
||||
|
||||
# Import conversations
|
||||
python scripts/import-conversations.py
|
||||
|
||||
# Archive files
|
||||
python scripts/archive-imported-conversations.py --skip-verification
|
||||
|
||||
# Check system health
|
||||
python scripts/check-tombstones.py
|
||||
bash scripts/test-context-recall.sh
|
||||
|
||||
# Database connection test
|
||||
python scripts/test_db_connection.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Requirements
|
||||
|
||||
Most scripts require:
|
||||
- Python 3.8+
|
||||
- Virtual environment activated (`api\venv\Scripts\activate`)
|
||||
- `.env` file configured (see `.env.example`)
|
||||
- Database access (172.16.3.30:3306)
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
|
||||
Scripts use these from `.env`:
|
||||
|
||||
```bash
|
||||
DATABASE_URL=mysql+pymysql://user:pass@172.16.3.30:3306/claudetools
|
||||
JWT_TOKEN=your-jwt-token-here
|
||||
API_USER_EMAIL=user@example.com
|
||||
API_USER_PASSWORD=your-password
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
- `TOMBSTONE_QUICK_START.md` - Tombstone system quick start
|
||||
- `../TOMBSTONE_SYSTEM.md` - Complete tombstone documentation
|
||||
- `../.claude/CONTEXT_RECALL_QUICK_START.md` - Context recall guide
|
||||
- `../CONTEXT_RECALL_SETUP.md` - Full setup instructions
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-01-18
|
||||
**Version:** 1.0
|
||||
@@ -1,311 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Direct Database Import Script
|
||||
|
||||
Imports Claude conversation contexts directly to the database,
|
||||
bypassing the API. Useful when the API is on a remote server
|
||||
but the conversation files are local.
|
||||
|
||||
Usage:
|
||||
python scripts/direct_db_import.py --folder "C:\\Users\\MikeSwanson\\.claude\\projects" --dry-run
|
||||
python scripts/direct_db_import.py --folder "C:\\Users\\MikeSwanson\\.claude\\projects" --execute
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Add parent directory to path for imports
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
from api.utils.conversation_parser import (
|
||||
extract_context_from_conversation,
|
||||
parse_jsonl_conversation,
|
||||
scan_folder_for_conversations,
|
||||
)
|
||||
from api.models.conversation_context import ConversationContext
|
||||
from api.schemas.conversation_context import ConversationContextCreate
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
|
||||
|
||||
def get_database_url():
|
||||
"""Get database URL from environment."""
|
||||
# Load from .env file
|
||||
env_path = Path(__file__).parent.parent / ".env"
|
||||
if env_path.exists():
|
||||
load_dotenv(env_path)
|
||||
|
||||
db_url = os.getenv("DATABASE_URL")
|
||||
if not db_url:
|
||||
print("[ERROR] DATABASE_URL not found in .env file")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"[OK] Database: {db_url.split('@')[1] if '@' in db_url else 'configured'}")
|
||||
return db_url
|
||||
|
||||
|
||||
def import_conversations(folder_path: str, dry_run: bool = True, project_id: str = None):
|
||||
"""
|
||||
Import conversations directly to database.
|
||||
|
||||
Args:
|
||||
folder_path: Path to folder containing .jsonl files
|
||||
dry_run: If True, preview without saving
|
||||
project_id: Optional project ID to associate contexts with
|
||||
"""
|
||||
print("\n" + "=" * 70)
|
||||
print("DIRECT DATABASE IMPORT")
|
||||
print("=" * 70)
|
||||
print(f"Mode: {'DRY RUN (preview only)' if dry_run else 'EXECUTE (will save to database)'}")
|
||||
print(f"Folder: {folder_path}")
|
||||
print("")
|
||||
|
||||
# Results tracking
|
||||
result = {
|
||||
"files_scanned": 0,
|
||||
"files_processed": 0,
|
||||
"contexts_created": 0,
|
||||
"errors": [],
|
||||
"contexts_preview": [],
|
||||
"contexts_data": [], # Store full context data for database insert
|
||||
}
|
||||
|
||||
# Step 1: Scan for conversation files
|
||||
print("[1/3] Scanning folder for conversation files...")
|
||||
try:
|
||||
conversation_files = scan_folder_for_conversations(folder_path)
|
||||
result["files_scanned"] = len(conversation_files)
|
||||
print(f" Found {len(conversation_files)} .jsonl files")
|
||||
except Exception as e:
|
||||
print(f"[ERROR] Failed to scan folder: {e}")
|
||||
return result
|
||||
|
||||
if not conversation_files:
|
||||
print("[WARNING] No conversation files found")
|
||||
return result
|
||||
|
||||
# Step 2: Parse conversations
|
||||
print(f"\n[2/3] Parsing conversations...")
|
||||
print(f"[DEBUG] dry_run = {dry_run}")
|
||||
|
||||
for file_path in conversation_files:
|
||||
try:
|
||||
# Parse conversation
|
||||
conversation = parse_jsonl_conversation(file_path)
|
||||
|
||||
if not conversation.get("messages"):
|
||||
result["errors"].append({
|
||||
"file": file_path,
|
||||
"error": "No messages found"
|
||||
})
|
||||
continue
|
||||
|
||||
# Extract context
|
||||
raw_context = extract_context_from_conversation(conversation)
|
||||
|
||||
# Transform to database format
|
||||
metadata = raw_context.get("raw_metadata", {})
|
||||
summary_obj = raw_context.get("summary", {})
|
||||
|
||||
# Get title from metadata or generate one
|
||||
title = metadata.get("title") or metadata.get("conversation_id") or f"Conversation"
|
||||
|
||||
# Get dense summary from summary object
|
||||
dense_summary = summary_obj.get("summary") or summary_obj.get("dense_summary") or "No summary available"
|
||||
|
||||
# Transform context to database format
|
||||
import json
|
||||
|
||||
# Convert decisions and tags to JSON strings
|
||||
decisions = raw_context.get("decisions", [])
|
||||
key_decisions_json = json.dumps(decisions) if decisions else None
|
||||
|
||||
tags = raw_context.get("tags", [])
|
||||
tags_json = json.dumps(tags) if tags else None
|
||||
|
||||
context = {
|
||||
"project_id": project_id,
|
||||
"session_id": None,
|
||||
"machine_id": None,
|
||||
"context_type": raw_context.get("category", "general_context"),
|
||||
"title": title,
|
||||
"dense_summary": dense_summary,
|
||||
"key_decisions": key_decisions_json,
|
||||
"current_state": None,
|
||||
"tags": tags_json,
|
||||
"relevance_score": raw_context.get("metrics", {}).get("quality_score", 5.0),
|
||||
}
|
||||
|
||||
result["files_processed"] += 1
|
||||
result["contexts_preview"].append({
|
||||
"title": context["title"],
|
||||
"type": context["context_type"],
|
||||
"message_count": len(conversation["messages"]),
|
||||
"tags": context.get("tags", []),
|
||||
"relevance_score": context.get("relevance_score", 0.0),
|
||||
})
|
||||
|
||||
# Store full context data for database insert
|
||||
if not dry_run:
|
||||
result["contexts_data"].append(context)
|
||||
print(f" [DEBUG] Stored context: {context['title'][:50]}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"[DEBUG] Exception for {Path(file_path).name}: {e}")
|
||||
result["errors"].append({
|
||||
"file": file_path,
|
||||
"error": str(e)
|
||||
})
|
||||
|
||||
print(f" Processed {result['files_processed']} files successfully")
|
||||
print(f" Errors: {len(result['errors'])}")
|
||||
|
||||
# Step 3: Save to database (if execute mode)
|
||||
if not dry_run:
|
||||
print(f"\n[3/3] Saving to database...")
|
||||
|
||||
try:
|
||||
# Create database connection
|
||||
db_url = get_database_url()
|
||||
engine = create_engine(db_url)
|
||||
SessionLocal = sessionmaker(bind=engine)
|
||||
db = SessionLocal()
|
||||
|
||||
# Save each context
|
||||
saved_count = 0
|
||||
for context_data in result["contexts_data"]:
|
||||
try:
|
||||
# Create context object
|
||||
context_obj = ConversationContext(
|
||||
project_id=context_data.get("project_id"),
|
||||
session_id=context_data.get("session_id"),
|
||||
machine_id=context_data.get("machine_id"),
|
||||
context_type=context_data["context_type"],
|
||||
title=context_data["title"],
|
||||
dense_summary=context_data["dense_summary"],
|
||||
key_decisions=context_data.get("key_decisions"),
|
||||
current_state=context_data.get("current_state"),
|
||||
tags=context_data.get("tags", []),
|
||||
relevance_score=context_data.get("relevance_score", 5.0),
|
||||
)
|
||||
|
||||
db.add(context_obj)
|
||||
saved_count += 1
|
||||
|
||||
except Exception as e:
|
||||
print(f"[WARNING] Failed to save context '{context_data.get('title', 'Unknown')}': {e}")
|
||||
|
||||
# Commit all changes
|
||||
db.commit()
|
||||
db.close()
|
||||
|
||||
result["contexts_created"] = saved_count
|
||||
print(f" Saved {saved_count} contexts to database")
|
||||
|
||||
except Exception as e:
|
||||
print(f"[ERROR] Database error: {e}")
|
||||
return result
|
||||
else:
|
||||
print(f"\n[3/3] Skipping database save (dry run mode)")
|
||||
|
||||
# Display results
|
||||
print("\n" + "=" * 70)
|
||||
print("IMPORT RESULTS")
|
||||
print("=" * 70)
|
||||
print(f"\nFiles scanned: {result['files_scanned']}")
|
||||
print(f"Files processed: {result['files_processed']}")
|
||||
print(f"Contexts created: {result['contexts_created'] if not dry_run else 'N/A (dry run)'}")
|
||||
print(f"Errors: {len(result['errors'])}")
|
||||
|
||||
# Show preview of contexts
|
||||
if result["contexts_preview"]:
|
||||
print(f"\n[PREVIEW] First 5 contexts:")
|
||||
for i, ctx in enumerate(result["contexts_preview"][:5], 1):
|
||||
print(f"\n {i}. {ctx['title']}")
|
||||
print(f" Type: {ctx['type']}")
|
||||
print(f" Messages: {ctx['message_count']}")
|
||||
print(f" Tags: {', '.join(ctx.get('tags', [])[:5])}")
|
||||
print(f" Relevance: {ctx.get('relevance_score', 0.0):.1f}/10.0")
|
||||
|
||||
# Show errors
|
||||
if result["errors"]:
|
||||
print(f"\n[ERRORS] First 5 errors:")
|
||||
for i, err in enumerate(result["errors"][:5], 1):
|
||||
print(f"\n {i}. File: {Path(err['file']).name}")
|
||||
print(f" Error: {err['error']}")
|
||||
|
||||
if len(result["errors"]) > 5:
|
||||
print(f"\n ... and {len(result['errors']) - 5} more errors")
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
"""Main entry point."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Import Claude conversations directly to database (bypasses API)"
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--folder",
|
||||
required=True,
|
||||
help="Path to folder containing .jsonl conversation files"
|
||||
)
|
||||
|
||||
mode_group = parser.add_mutually_exclusive_group(required=True)
|
||||
mode_group.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="Preview import without saving"
|
||||
)
|
||||
mode_group.add_argument(
|
||||
"--execute",
|
||||
action="store_true",
|
||||
help="Execute import and save to database"
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--project-id",
|
||||
help="Associate all contexts with this project ID"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Validate folder
|
||||
folder_path = Path(args.folder)
|
||||
if not folder_path.exists():
|
||||
print(f"[ERROR] Folder does not exist: {folder_path}")
|
||||
sys.exit(1)
|
||||
|
||||
# Run import
|
||||
try:
|
||||
result = import_conversations(
|
||||
folder_path=str(folder_path),
|
||||
dry_run=args.dry_run,
|
||||
project_id=args.project_id
|
||||
)
|
||||
|
||||
# Success message
|
||||
if args.dry_run:
|
||||
print("\n[SUCCESS] Dry run completed")
|
||||
print(" Run with --execute to save to database")
|
||||
else:
|
||||
print(f"\n[SUCCESS] Import completed")
|
||||
print(f" Created {result['contexts_created']} contexts")
|
||||
|
||||
sys.exit(0)
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n[ERROR] Import failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
411
scripts/export-tombstoned-contexts.py
Normal file
411
scripts/export-tombstoned-contexts.py
Normal file
@@ -0,0 +1,411 @@
|
||||
"""
|
||||
Export Tombstoned Contexts Before Removal
|
||||
|
||||
This script exports all conversation contexts referenced by tombstone files
|
||||
and any additional contexts in the database to markdown files before the
|
||||
context system is removed from ClaudeTools.
|
||||
|
||||
Features:
|
||||
- Finds all *.tombstone.json files
|
||||
- Extracts context_ids from tombstones
|
||||
- Retrieves contexts from database via API
|
||||
- Exports to markdown files organized by project/date
|
||||
- Handles cases where no tombstones or contexts exist
|
||||
|
||||
Usage:
|
||||
# Export all tombstoned contexts
|
||||
python scripts/export-tombstoned-contexts.py
|
||||
|
||||
# Specify custom output directory
|
||||
python scripts/export-tombstoned-contexts.py --output exported-contexts
|
||||
|
||||
# Include all database contexts (not just tombstoned ones)
|
||||
python scripts/export-tombstoned-contexts.py --export-all
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Any
|
||||
|
||||
import requests
|
||||
from dotenv import load_dotenv
|
||||
import os
|
||||
|
||||
|
||||
# Constants
|
||||
DEFAULT_API_URL = "http://172.16.3.30:8001"
|
||||
DEFAULT_OUTPUT_DIR = Path("D:/ClaudeTools/exported-contexts")
|
||||
IMPORTED_CONVERSATIONS_DIR = Path("D:/ClaudeTools/imported-conversations")
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
|
||||
def print_status(message: str, status: str = "INFO") -> None:
|
||||
"""Print formatted status message."""
|
||||
markers = {
|
||||
"INFO": "[INFO]",
|
||||
"SUCCESS": "[OK]",
|
||||
"WARNING": "[WARNING]",
|
||||
"ERROR": "[ERROR]"
|
||||
}
|
||||
print(f"{markers.get(status, '[INFO]')} {message}")
|
||||
|
||||
|
||||
def get_jwt_token(api_url: str) -> Optional[str]:
|
||||
"""
|
||||
Get JWT token from environment or API.
|
||||
|
||||
Args:
|
||||
api_url: Base URL for API
|
||||
|
||||
Returns:
|
||||
JWT token or None if failed
|
||||
"""
|
||||
token = os.getenv("JWT_TOKEN")
|
||||
if token:
|
||||
return token
|
||||
|
||||
email = os.getenv("API_USER_EMAIL", "admin@claudetools.local")
|
||||
password = os.getenv("API_USER_PASSWORD", "claudetools123")
|
||||
|
||||
try:
|
||||
response = requests.post(
|
||||
f"{api_url}/api/auth/token",
|
||||
data={"username": email, "password": password}
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.json()["access_token"]
|
||||
except Exception as e:
|
||||
print_status(f"Failed to get JWT token: {e}", "ERROR")
|
||||
return None
|
||||
|
||||
|
||||
def find_tombstone_files(base_dir: Path) -> List[Path]:
|
||||
"""Find all tombstone files."""
|
||||
if not base_dir.exists():
|
||||
return []
|
||||
return sorted(base_dir.rglob("*.tombstone.json"))
|
||||
|
||||
|
||||
def extract_context_ids_from_tombstones(tombstone_files: List[Path]) -> List[str]:
|
||||
"""
|
||||
Extract all context IDs from tombstone files.
|
||||
|
||||
Args:
|
||||
tombstone_files: List of tombstone file paths
|
||||
|
||||
Returns:
|
||||
List of unique context IDs
|
||||
"""
|
||||
context_ids = set()
|
||||
|
||||
for tombstone_path in tombstone_files:
|
||||
try:
|
||||
with open(tombstone_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
ids = data.get("context_ids", [])
|
||||
context_ids.update(ids)
|
||||
except Exception as e:
|
||||
print_status(f"Failed to read {tombstone_path.name}: {e}", "WARNING")
|
||||
|
||||
return list(context_ids)
|
||||
|
||||
|
||||
def fetch_context_from_api(
|
||||
context_id: str,
|
||||
api_url: str,
|
||||
jwt_token: str
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Fetch a single context from the API.
|
||||
|
||||
Args:
|
||||
context_id: Context UUID
|
||||
api_url: API base URL
|
||||
jwt_token: JWT authentication token
|
||||
|
||||
Returns:
|
||||
Context data dict or None if failed
|
||||
"""
|
||||
try:
|
||||
headers = {"Authorization": f"Bearer {jwt_token}"}
|
||||
response = requests.get(
|
||||
f"{api_url}/api/conversation-contexts/{context_id}",
|
||||
headers=headers
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
return response.json()
|
||||
elif response.status_code == 404:
|
||||
print_status(f"Context {context_id} not found in database", "WARNING")
|
||||
else:
|
||||
print_status(f"Failed to fetch context {context_id}: HTTP {response.status_code}", "WARNING")
|
||||
|
||||
except Exception as e:
|
||||
print_status(f"Error fetching context {context_id}: {e}", "WARNING")
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def fetch_all_contexts(api_url: str, jwt_token: str) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Fetch all contexts from the API.
|
||||
|
||||
Args:
|
||||
api_url: API base URL
|
||||
jwt_token: JWT authentication token
|
||||
|
||||
Returns:
|
||||
List of context data dicts
|
||||
"""
|
||||
contexts = []
|
||||
headers = {"Authorization": f"Bearer {jwt_token}"}
|
||||
|
||||
try:
|
||||
# Fetch paginated results
|
||||
offset = 0
|
||||
limit = 100
|
||||
|
||||
while True:
|
||||
response = requests.get(
|
||||
f"{api_url}/api/conversation-contexts",
|
||||
headers=headers,
|
||||
params={"offset": offset, "limit": limit}
|
||||
)
|
||||
|
||||
if response.status_code != 200:
|
||||
print_status(f"Failed to fetch contexts: HTTP {response.status_code}", "ERROR")
|
||||
break
|
||||
|
||||
data = response.json()
|
||||
|
||||
# Handle different response formats
|
||||
if isinstance(data, list):
|
||||
batch = data
|
||||
elif isinstance(data, dict) and "items" in data:
|
||||
batch = data["items"]
|
||||
else:
|
||||
batch = []
|
||||
|
||||
if not batch:
|
||||
break
|
||||
|
||||
contexts.extend(batch)
|
||||
offset += len(batch)
|
||||
|
||||
# Check if we've fetched all
|
||||
if len(batch) < limit:
|
||||
break
|
||||
|
||||
except Exception as e:
|
||||
print_status(f"Error fetching all contexts: {e}", "ERROR")
|
||||
|
||||
return contexts
|
||||
|
||||
|
||||
def export_context_to_markdown(
|
||||
context: Dict[str, Any],
|
||||
output_dir: Path
|
||||
) -> Optional[Path]:
|
||||
"""
|
||||
Export a single context to a markdown file.
|
||||
|
||||
Args:
|
||||
context: Context data dict
|
||||
output_dir: Output directory
|
||||
|
||||
Returns:
|
||||
Path to exported file or None if failed
|
||||
"""
|
||||
try:
|
||||
# Extract context data
|
||||
context_id = context.get("id", "unknown")
|
||||
title = context.get("title", "Untitled")
|
||||
context_type = context.get("context_type", "unknown")
|
||||
created_at = context.get("created_at", "unknown")
|
||||
|
||||
# Parse date for organization
|
||||
try:
|
||||
dt = datetime.fromisoformat(created_at.replace("Z", "+00:00"))
|
||||
date_dir = output_dir / dt.strftime("%Y-%m")
|
||||
except:
|
||||
date_dir = output_dir / "undated"
|
||||
|
||||
date_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Create safe filename
|
||||
safe_title = "".join(c if c.isalnum() or c in (' ', '-', '_') else '_' for c in title)
|
||||
safe_title = safe_title[:50] # Limit length
|
||||
filename = f"{context_id[:8]}_{safe_title}.md"
|
||||
output_path = date_dir / filename
|
||||
|
||||
# Build markdown content
|
||||
markdown = f"""# {title}
|
||||
|
||||
**Type:** {context_type}
|
||||
**Created:** {created_at}
|
||||
**Context ID:** {context_id}
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
{context.get('dense_summary', 'No summary available')}
|
||||
|
||||
---
|
||||
|
||||
## Key Decisions
|
||||
|
||||
{context.get('key_decisions', 'No key decisions recorded')}
|
||||
|
||||
---
|
||||
|
||||
## Current State
|
||||
|
||||
{context.get('current_state', 'No current state recorded')}
|
||||
|
||||
---
|
||||
|
||||
## Tags
|
||||
|
||||
{context.get('tags', 'No tags')}
|
||||
|
||||
---
|
||||
|
||||
## Metadata
|
||||
|
||||
- **Session ID:** {context.get('session_id', 'N/A')}
|
||||
- **Project ID:** {context.get('project_id', 'N/A')}
|
||||
- **Machine ID:** {context.get('machine_id', 'N/A')}
|
||||
- **Relevance Score:** {context.get('relevance_score', 'N/A')}
|
||||
|
||||
---
|
||||
|
||||
*Exported on {datetime.now().isoformat()}*
|
||||
"""
|
||||
|
||||
# Write to file
|
||||
with open(output_path, "w", encoding="utf-8") as f:
|
||||
f.write(markdown)
|
||||
|
||||
return output_path
|
||||
|
||||
except Exception as e:
|
||||
print_status(f"Failed to export context {context.get('id', 'unknown')}: {e}", "ERROR")
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
"""Main entry point."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Export tombstoned contexts before removal"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=Path,
|
||||
default=DEFAULT_OUTPUT_DIR,
|
||||
help=f"Output directory (default: {DEFAULT_OUTPUT_DIR})"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--api-url",
|
||||
default=DEFAULT_API_URL,
|
||||
help=f"API base URL (default: {DEFAULT_API_URL})"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--export-all",
|
||||
action="store_true",
|
||||
help="Export ALL database contexts, not just tombstoned ones"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
print_status("=" * 80, "INFO")
|
||||
print_status("ClaudeTools Context Export Tool", "INFO")
|
||||
print_status("=" * 80, "INFO")
|
||||
print_status(f"Output directory: {args.output}", "INFO")
|
||||
print_status(f"Export all contexts: {'YES' if args.export_all else 'NO'}", "INFO")
|
||||
print_status("=" * 80, "INFO")
|
||||
|
||||
# Create output directory
|
||||
args.output.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Get JWT token
|
||||
print_status("\nAuthenticating with API...", "INFO")
|
||||
jwt_token = get_jwt_token(args.api_url)
|
||||
if not jwt_token:
|
||||
print_status("Cannot proceed without API access", "ERROR")
|
||||
sys.exit(1)
|
||||
|
||||
print_status("Authentication successful", "SUCCESS")
|
||||
|
||||
# Find tombstone files
|
||||
print_status("\nSearching for tombstone files...", "INFO")
|
||||
tombstone_files = find_tombstone_files(IMPORTED_CONVERSATIONS_DIR)
|
||||
print_status(f"Found {len(tombstone_files)} tombstone files", "INFO")
|
||||
|
||||
# Extract context IDs from tombstones
|
||||
context_ids = []
|
||||
if tombstone_files:
|
||||
print_status("\nExtracting context IDs from tombstones...", "INFO")
|
||||
context_ids = extract_context_ids_from_tombstones(tombstone_files)
|
||||
print_status(f"Found {len(context_ids)} unique context IDs in tombstones", "INFO")
|
||||
|
||||
# Fetch contexts
|
||||
contexts = []
|
||||
|
||||
if args.export_all:
|
||||
print_status("\nFetching ALL contexts from database...", "INFO")
|
||||
contexts = fetch_all_contexts(args.api_url, jwt_token)
|
||||
print_status(f"Retrieved {len(contexts)} total contexts", "INFO")
|
||||
elif context_ids:
|
||||
print_status("\nFetching tombstoned contexts from database...", "INFO")
|
||||
for i, context_id in enumerate(context_ids, 1):
|
||||
print_status(f"Fetching context {i}/{len(context_ids)}: {context_id}", "INFO")
|
||||
context = fetch_context_from_api(context_id, args.api_url, jwt_token)
|
||||
if context:
|
||||
contexts.append(context)
|
||||
print_status(f"Successfully retrieved {len(contexts)} contexts", "INFO")
|
||||
else:
|
||||
print_status("\nNo tombstone files found and --export-all not specified", "WARNING")
|
||||
print_status("Attempting to fetch all database contexts anyway...", "INFO")
|
||||
contexts = fetch_all_contexts(args.api_url, jwt_token)
|
||||
if contexts:
|
||||
print_status(f"Retrieved {len(contexts)} contexts from database", "INFO")
|
||||
|
||||
# Export contexts to markdown
|
||||
if not contexts:
|
||||
print_status("\nNo contexts to export", "WARNING")
|
||||
print_status("This is normal if the context system was never used", "INFO")
|
||||
return
|
||||
|
||||
print_status(f"\nExporting {len(contexts)} contexts to markdown...", "INFO")
|
||||
exported_count = 0
|
||||
|
||||
for i, context in enumerate(contexts, 1):
|
||||
print_status(f"Exporting {i}/{len(contexts)}: {context.get('title', 'Untitled')}", "INFO")
|
||||
output_path = export_context_to_markdown(context, args.output)
|
||||
if output_path:
|
||||
exported_count += 1
|
||||
|
||||
# Summary
|
||||
print_status("\n" + "=" * 80, "INFO")
|
||||
print_status("EXPORT SUMMARY", "INFO")
|
||||
print_status("=" * 80, "INFO")
|
||||
print_status(f"Tombstone files found: {len(tombstone_files)}", "INFO")
|
||||
print_status(f"Contexts retrieved: {len(contexts)}", "INFO")
|
||||
print_status(f"Contexts exported: {exported_count}", "SUCCESS")
|
||||
print_status(f"Output directory: {args.output}", "INFO")
|
||||
print_status("=" * 80, "INFO")
|
||||
|
||||
if exported_count > 0:
|
||||
print_status(f"\n[SUCCESS] Exported {exported_count} contexts to {args.output}", "SUCCESS")
|
||||
else:
|
||||
print_status("\n[WARNING] No contexts were exported", "WARNING")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,284 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Claude Context Import Script
|
||||
|
||||
Command-line tool to bulk import conversation contexts from Claude project folders.
|
||||
|
||||
Usage:
|
||||
python scripts/import-claude-context.py --folder "C:/Users/MikeSwanson/claude-projects" --dry-run
|
||||
python scripts/import-claude-context.py --folder "C:/Users/MikeSwanson/claude-projects" --execute
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
from dotenv import load_dotenv
|
||||
|
||||
|
||||
def load_jwt_token() -> str:
|
||||
"""
|
||||
Load JWT token from .claude/context-recall-config.env
|
||||
|
||||
Returns:
|
||||
JWT token string
|
||||
|
||||
Raises:
|
||||
SystemExit: If token cannot be loaded
|
||||
"""
|
||||
# Try multiple possible locations
|
||||
possible_paths = [
|
||||
Path(".claude/context-recall-config.env"),
|
||||
Path("D:/ClaudeTools/.claude/context-recall-config.env"),
|
||||
Path(__file__).parent.parent / ".claude" / "context-recall-config.env",
|
||||
]
|
||||
|
||||
for env_path in possible_paths:
|
||||
if env_path.exists():
|
||||
load_dotenv(env_path)
|
||||
token = os.getenv("JWT_TOKEN")
|
||||
if token:
|
||||
print(f"[OK] Loaded JWT token from {env_path}")
|
||||
return token
|
||||
|
||||
print("[ERROR] Could not find JWT_TOKEN in .claude/context-recall-config.env")
|
||||
print("\nTried locations:")
|
||||
for path in possible_paths:
|
||||
print(f" - {path} ({'exists' if path.exists() else 'not found'})")
|
||||
print("\nPlease create .claude/context-recall-config.env with:")
|
||||
print(" JWT_TOKEN=your_token_here")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def get_api_base_url() -> str:
|
||||
"""
|
||||
Get API base URL from environment or use default.
|
||||
|
||||
Returns:
|
||||
API base URL string
|
||||
"""
|
||||
return os.getenv("API_BASE_URL", "http://localhost:8000")
|
||||
|
||||
|
||||
def call_bulk_import_api(
|
||||
folder_path: str,
|
||||
jwt_token: str,
|
||||
dry_run: bool = True,
|
||||
project_id: str = None,
|
||||
session_id: str = None,
|
||||
) -> dict:
|
||||
"""
|
||||
Call the bulk import API endpoint.
|
||||
|
||||
Args:
|
||||
folder_path: Path to folder containing Claude conversations
|
||||
jwt_token: JWT authentication token
|
||||
dry_run: Preview mode without saving
|
||||
project_id: Optional project ID to associate contexts with
|
||||
session_id: Optional session ID to associate contexts with
|
||||
|
||||
Returns:
|
||||
API response dictionary
|
||||
|
||||
Raises:
|
||||
requests.exceptions.RequestException: If API call fails
|
||||
"""
|
||||
api_url = f"{get_api_base_url()}/api/bulk-import/import-folder"
|
||||
|
||||
headers = {
|
||||
"Authorization": f"Bearer {jwt_token}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
params = {
|
||||
"folder_path": folder_path,
|
||||
"dry_run": dry_run,
|
||||
}
|
||||
|
||||
if project_id:
|
||||
params["project_id"] = project_id
|
||||
if session_id:
|
||||
params["session_id"] = session_id
|
||||
|
||||
print(f"\n[API] Calling: {api_url}")
|
||||
print(f" Mode: {'DRY RUN' if dry_run else 'EXECUTE'}")
|
||||
print(f" Folder: {folder_path}")
|
||||
|
||||
response = requests.post(api_url, headers=headers, params=params, timeout=300)
|
||||
response.raise_for_status()
|
||||
|
||||
return response.json()
|
||||
|
||||
|
||||
def display_progress(result: dict):
|
||||
"""
|
||||
Display import progress and results.
|
||||
|
||||
Args:
|
||||
result: API response dictionary
|
||||
"""
|
||||
print("\n" + "=" * 70)
|
||||
print("IMPORT RESULTS")
|
||||
print("=" * 70)
|
||||
|
||||
# Summary
|
||||
print(f"\n{result.get('summary', 'No summary available')}")
|
||||
|
||||
# Statistics
|
||||
print(f"\n[STATS]")
|
||||
print(f" Files scanned: {result.get('files_scanned', 0)}")
|
||||
print(f" Files processed: {result.get('files_processed', 0)}")
|
||||
print(f" Contexts created: {result.get('contexts_created', 0)}")
|
||||
print(f" Errors: {len(result.get('errors', []))}")
|
||||
|
||||
# Context preview
|
||||
contexts_preview = result.get("contexts_preview", [])
|
||||
if contexts_preview:
|
||||
print(f"\n[PREVIEW] Contexts (showing {min(5, len(contexts_preview))} of {len(contexts_preview)}):")
|
||||
for i, ctx in enumerate(contexts_preview[:5], 1):
|
||||
print(f"\n {i}. {ctx.get('title', 'Untitled')}")
|
||||
print(f" Type: {ctx.get('type', 'unknown')}")
|
||||
print(f" Messages: {ctx.get('message_count', 0)}")
|
||||
print(f" Tags: {', '.join(ctx.get('tags', []))}")
|
||||
print(f" Relevance: {ctx.get('relevance_score', 0.0):.1f}/10.0")
|
||||
|
||||
# Errors
|
||||
errors = result.get("errors", [])
|
||||
if errors:
|
||||
print(f"\n[WARNING] Errors ({len(errors)}):")
|
||||
for i, error in enumerate(errors[:5], 1):
|
||||
print(f"\n {i}. File: {error.get('file', 'unknown')}")
|
||||
print(f" Error: {error.get('error', 'unknown error')}")
|
||||
if len(errors) > 5:
|
||||
print(f"\n ... and {len(errors) - 5} more errors")
|
||||
|
||||
print("\n" + "=" * 70)
|
||||
|
||||
|
||||
def main():
|
||||
"""Main entry point for the import script."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Import Claude conversation contexts from project folders",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
# Preview import without saving
|
||||
python scripts/import-claude-context.py --folder "C:\\Users\\MikeSwanson\\claude-projects" --dry-run
|
||||
|
||||
# Execute import and save to database
|
||||
python scripts/import-claude-context.py --folder "C:\\Users\\MikeSwanson\\claude-projects" --execute
|
||||
|
||||
# Associate with a specific project
|
||||
python scripts/import-claude-context.py --folder "C:\\Users\\MikeSwanson\\claude-projects" --execute --project-id abc-123
|
||||
"""
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--folder",
|
||||
required=True,
|
||||
help="Path to Claude projects folder containing .jsonl conversation files"
|
||||
)
|
||||
|
||||
mode_group = parser.add_mutually_exclusive_group(required=True)
|
||||
mode_group.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="Preview import without saving to database"
|
||||
)
|
||||
mode_group.add_argument(
|
||||
"--execute",
|
||||
action="store_true",
|
||||
help="Execute import and save to database"
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--project-id",
|
||||
help="Associate all imported contexts with this project ID"
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--session-id",
|
||||
help="Associate all imported contexts with this session ID"
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--api-url",
|
||||
help="API base URL (default: http://localhost:8000)"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Set API URL if provided
|
||||
if args.api_url:
|
||||
os.environ["API_BASE_URL"] = args.api_url
|
||||
|
||||
# Validate folder path
|
||||
folder_path = Path(args.folder)
|
||||
if not folder_path.exists():
|
||||
print(f"[ERROR] Folder does not exist: {folder_path}")
|
||||
sys.exit(1)
|
||||
|
||||
print("=" * 70)
|
||||
print("CLAUDE CONTEXT IMPORT TOOL")
|
||||
print("=" * 70)
|
||||
|
||||
# Load JWT token
|
||||
try:
|
||||
jwt_token = load_jwt_token()
|
||||
except Exception as e:
|
||||
print(f"[ERROR] Error loading JWT token: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
# Determine mode
|
||||
dry_run = args.dry_run
|
||||
|
||||
# Call API
|
||||
try:
|
||||
result = call_bulk_import_api(
|
||||
folder_path=str(folder_path),
|
||||
jwt_token=jwt_token,
|
||||
dry_run=dry_run,
|
||||
project_id=args.project_id,
|
||||
session_id=args.session_id,
|
||||
)
|
||||
|
||||
# Display results
|
||||
display_progress(result)
|
||||
|
||||
# Success message
|
||||
if dry_run:
|
||||
print("\n[SUCCESS] Dry run completed successfully!")
|
||||
print(" Run with --execute to save contexts to database")
|
||||
else:
|
||||
print(f"\n[SUCCESS] Import completed successfully!")
|
||||
print(f" Created {result.get('contexts_created', 0)} contexts")
|
||||
|
||||
sys.exit(0)
|
||||
|
||||
except requests.exceptions.HTTPError as e:
|
||||
print(f"\n[ERROR] API Error: {e}")
|
||||
if e.response is not None:
|
||||
try:
|
||||
error_detail = e.response.json()
|
||||
print(f" Detail: {error_detail.get('detail', 'No details available')}")
|
||||
except:
|
||||
print(f" Response: {e.response.text}")
|
||||
sys.exit(1)
|
||||
|
||||
except requests.exceptions.RequestException as e:
|
||||
print(f"\n[ERROR] Network Error: {e}")
|
||||
print(" Make sure the API server is running")
|
||||
sys.exit(1)
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n[ERROR] Unexpected Error: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user