claudetools/.claude/FILE_ORGANIZATION.md

# File Organization & Storage Strategy

## Overview

The ClaudeTools system uses a **hybrid storage approach**:
- **Database** - Metadata, context, task status, relationships, insights
- **Filesystem** - Actual code files, documentation, scripts, configurations
- **Gitea** - Version control for all file-based work
- **Backups** - Local database dumps and file backups

## Storage Philosophy

### What Goes in Database (via Database Agent)
- Task and session metadata
- Work item records (problem/solution summaries)
- Client and infrastructure details
- Environmental insights
- Failure patterns
- Command history
- Credentials (encrypted)
- Relationships between entities
- Timestamps and audit trails

### What Goes on Filesystem (version controlled)
- Source code files
- Configuration files
- Documentation (markdown, txt)
- Scripts (PowerShell, Bash, Python)
- Diagrams and images
- README files
- Project-specific notes
- Session logs (markdown)

### Why Hybrid?
- **Database**: Fast queries, relationships, metadata
- **Filesystem**: Version control, diff tracking, editor-friendly
- **Gitea**: Backup, history, team collaboration
- **Best of both worlds**: Structured data + version-controlled files

## Folder Structure

```
D:\ClaudeTools\
├── .claude\                    # System configuration
│   ├── agents\                 # Agent definitions
│   │   ├── coding.md
│   │   ├── code-review.md
│   │   ├── database.md
│   │   ├── gitea.md
│   │   └── backup.md
│   ├── plans\                  # Plan mode outputs
│   ├── settings.local.json     # Local settings
│   ├── CODE_WORKFLOW.md
│   ├── TASK_MANAGEMENT.md
│   ├── FILE_ORGANIZATION.md
│   └── MSP-MODE-SPEC.md
│
├── clients\                    # MSP Mode - Client work
│   ├── dataforth\
│   │   ├── configs\            # Configuration files
│   │   │   ├── nas\
│   │   │   │   ├── smb.conf.overrides
│   │   │   │   └── sync-to-ad2.sh
│   │   │   ├── ad2\
│   │   │   └── dos-machines\
│   │   ├── docs\               # Documentation
│   │   │   ├── README.md
│   │   │   ├── NETWORK_TOPOLOGY.md
│   │   │   ├── CREDENTIALS.md (encrypted)
│   │   │   └── TROUBLESHOOTING.md
│   │   ├── scripts\            # Automation scripts
│   │   │   ├── Sync-FromNAS.ps1
│   │   │   ├── UPDATE.BAT
│   │   │   └── monitor-nas.ps1
│   │   ├── session-logs\       # Session logs for this client
│   │   │   ├── 2026-01-15-dos-update.md
│   │   │   └── 2026-01-14-wins-fix.md
│   │   ├── .git\               # Git repository
│   │   ├── .gitignore
│   │   └── README.md           # Client overview
│   │
│   ├── grabb\
│   │   ├── configs\
│   │   ├── docs\
│   │   ├── scripts\
│   │   ├── session-logs\
│   │   └── README.md
│   │
│   └── [other-clients]\
│
├── projects\                   # Development Mode - Dev projects
│   ├── gururmm\
│   │   ├── agent\              # Agent source code (Rust)
│   │   │   ├── src\
│   │   │   ├── Cargo.toml
│   │   │   └── README.md
│   │   ├── server\             # Server source code (Rust)
│   │   │   ├── src\
│   │   │   ├── Cargo.toml
│   │   │   └── README.md
│   │   ├── dashboard\          # Dashboard source code (React)
│   │   │   ├── src\
│   │   │   ├── package.json
│   │   │   └── README.md
│   │   ├── docs\               # Project documentation
│   │   │   ├── ARCHITECTURE.md
│   │   │   ├── API.md
│   │   │   └── DEPLOYMENT.md
│   │   ├── scripts\            # Build/deploy scripts
│   │   ├── session-logs\       # Development session logs
│   │   ├── .git\
│   │   └── README.md
│   │
│   ├── claudetools-api\        # ClaudeTools API project
│   │   ├── src\
│   │   ├── docs\
│   │   ├── tests\
│   │   ├── .git\
│   │   └── README.md
│   │
│   └── [other-projects]\
│
├── normal\                     # Normal Mode - General work
│   ├── research\               # Research notes
│   │   ├── rust-async-patterns.md
│   │   ├── docker-networking.md
│   │   └── ...
│   ├── experiments\            # Code experiments
│   │   ├── test-project-1\
│   │   └── ...
│   ├── notes\                  # General notes
│   └── session-logs\           # Normal mode session logs
│
├── backups\                    # Backup storage
│   ├── database\               # Database backups
│   │   ├── claudetools-2026-01-15-daily.sql.gz
│   │   ├── claudetools-2026-01-14-daily.sql.gz
│   │   └── ...
│   ├── files\                  # File backups (snapshots)
│   │   ├── clients-2026-01-15.tar.gz
│   │   └── ...
│   └── .gitignore              # Don't commit backups
│
└── README.md                   # ClaudeTools overview
```

## Gitea Integration

### Repository Structure

Each client and project has its own Git repository:

**Client Repositories:**
- `azcomputerguru/claudetools-client-dataforth`
- `azcomputerguru/claudetools-client-grabb`
- etc.

**Project Repositories:**
- `azcomputerguru/gururmm` (already exists)
- `azcomputerguru/claudetools-api`
- etc.

**System Repository:**
- `azcomputerguru/claudetools` (for .claude configs, agents, system docs)

### Commit Strategy

**When to Commit:**
1. **After completing tasks** - Task marked 'completed' in database
2. **After significant work** - Code review approved, feature complete
3. **End of session** - Session ending, preserve work
4. **On user request** - User explicitly asks to commit
5. **Periodic commits** - Every N tasks completed (configurable)

**Commit Message Format:**
```
[Mode] Brief description

Detailed context:
- Task: [task title]
- Changes: [list of changes]
- Duration: [time spent]
- Status: [completed/in-progress]

Files modified:
- path/to/file1.py
- path/to/file2.md
```

**Example Commit:**
```
[MSP:Dataforth] Fix WINS service on D2TESTNAS

Detailed context:
- Task: Troubleshoot WINS service failure
- Changes: Fixed smb.conf.overrides syntax error
- Duration: 45 minutes (billable)
- Status: Completed and verified

Files modified:
- configs/nas/smb.conf.overrides
- docs/TROUBLESHOOTING.md
- session-logs/2026-01-15-wins-fix.md

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
```

### Git Workflow

**Orchestrator triggers Git operations via Gitea Agent:**

1. **Detect changes** - Files modified during work
2. **Stage changes** - `git add` modified files
3. **Create commit** - With context-rich message
4. **Push to Gitea** - Sync to remote repository
5. **Update database** - Record commit hash in session

## File Creation Rules

### Client Files (MSP Mode)

When working on client project:

1. **Determine client** - From session context
2. **Create/update files** in `D:\ClaudeTools\clients\[client-name]\`
3. **Use appropriate subdirectory**:
   - `configs/` - Configuration files
   - `docs/` - Documentation
   - `scripts/` - Automation scripts
   - `session-logs/` - Session logs
4. **Update README.md** - Keep client overview current
5. **Stage for commit** - Gitea Agent handles commit

### Project Files (Development Mode)

When working on development project:

1. **Determine project** - From session context
2. **Create/update files** in `D:\ClaudeTools\projects\[project-name]\`
3. **Follow project structure** - Respect existing organization
4. **Update documentation** - Keep docs current
5. **Write tests** - Test files alongside code
6. **Stage for commit** - Gitea Agent handles commit

### Normal Mode Files

When in Normal Mode:

1. **Categorize work**:
   - Research → `normal/research/`
   - Experiments → `normal/experiments/`
   - Notes → `normal/notes/`
2. **Use descriptive filenames** - Include date if temporal
3. **Commit periodically** - Keep work safe

## Database Backup Strategy

### Backup Schedule

**Daily backups:**
- Automatic at 2:00 AM (if system running)
- Or on first session of the day if not run overnight

**On-demand backups:**
- Before major changes (schema updates)
- After significant data entry
- On user request

### Backup Format

**Filename:** `claudetools-YYYY-MM-DD-[type].sql.gz`
- `type` = daily, manual, pre-migration, etc.

**Contents:**
- Full database dump (all tables, all data)
- Compressed with gzip
- Includes schema and data
- Stored in `D:\ClaudeTools\backups\database\`

**Example:**
```
claudetools-2026-01-15-daily.sql.gz
claudetools-2026-01-15-pre-migration.sql.gz
claudetools-2026-01-14-daily.sql.gz
```

### Backup Retention

- **Keep last 7 daily backups**
- **Keep last 4 weekly backups** (Sunday backups)
- **Keep last 12 monthly backups** (1st of month backups)
- **Keep pre-migration backups indefinitely**
- **Automatic cleanup** - Backup Agent purges old backups

### Backup Verification

After each backup:
1. **Check file size** - Not zero, reasonable size
2. **Test gzip integrity** - `gzip -t file.sql.gz`
3. **Record in database** - Log backup in `backup_log` table
4. **Optional: Test restore** - Periodically restore to verify

## Session Logs

### Where They Go

**MSP Mode:**
- `D:\ClaudeTools\clients\[client]\session-logs\YYYY-MM-DD-[description].md`

**Development Mode:**
- `D:\ClaudeTools\projects\[project]\session-logs\YYYY-MM-DD-[description].md`

**Normal Mode:**
- `D:\ClaudeTools\normal\session-logs\YYYY-MM-DD-[description].md`

**Shared (cross-mode sessions):**
- `C:\Users\MikeSwanson\claude-projects\session-logs\YYYY-MM-DD-session.md` (existing location)
- Symlinks or cross-references in mode-specific locations

### Session Log Format

```markdown
# Session: [Title]

**Date:** 2026-01-15
**Mode:** MSP (Dataforth) / Development (GuruRMM) / Normal
**Duration:** 2.5 hours
**Billable:** Yes (MSP only)

## Summary

[Brief description of what was accomplished]

## Tasks Completed

- [✓] Task 1 (30 min)
- [✓] Task 2 (45 min)
- [✓] Task 3 (15 min)

## Work Items (MSP Mode)

### Problem
[Description]

### Cause
[Root cause]

### Solution
[What was done]

### Verification
[How it was tested]

## Files Modified

- `path/to/file1.py` - [what changed]
- `path/to/file2.md` - [what changed]

## Key Learnings

- [Environmental insights discovered]
- [Failures encountered and resolved]
- [Patterns identified]

## Next Steps

- [ ] Pending task 1
- [ ] Pending task 2

## References

- Related sessions: [links]
- Documentation: [links]
- Tickets: [ticket IDs if MSP mode]
```

## File Naming Conventions

### Session Logs
`YYYY-MM-DD-brief-description.md`
- Example: `2026-01-15-wins-service-fix.md`
- Example: `2026-01-14-update-bat-simplification.md`

### Configuration Files
`service-config.ext` or `SERVICE.CONF`
- Example: `smb.conf.overrides`
- Example: `sync-to-ad2.sh`
- Match existing conventions where applicable

### Scripts
`verb-noun.ext`
- Example: `Sync-FromNAS.ps1` (PowerShell PascalCase)
- Example: `monitor-nas.sh` (Bash kebab-case)
- Example: `deploy-app.py` (Python snake_case)

### Documentation
`SCREAMING_CASE.md` for important docs, `Title Case.md` for regular
- Example: `README.md`
- Example: `ARCHITECTURE.md`
- Example: `API Documentation.md`

## .gitignore Files

Each repository should have appropriate `.gitignore`:

**Client repositories:**
```gitignore
# Credentials (encrypted versions are OK, plaintext never)
**/CREDENTIALS.txt
**/*password*
**/*secret*

# Temporary files
*.tmp
*.log
*.bak

# OS files
.DS_Store
Thumbs.db
```

**Project repositories:**
```gitignore
# Dependencies
node_modules/
target/
venv/
__pycache__/

# Build outputs
dist/
build/
*.exe
*.dll

# IDE
.vscode/
.idea/
*.swp

# Environment
.env
.env.local
```

**ClaudeTools root:**
```gitignore
# Backups (local only)
backups/

# Local settings
.claude/settings.local.json

# Temporary
*.tmp
```

## Integration with Database

### File References in Database

Store file paths in database for context:

```python
# In task_context field:
{
  "files_modified": [
    "clients/dataforth/configs/nas/smb.conf.overrides",
    "clients/dataforth/docs/TROUBLESHOOTING.md"
  ],
  "files_created": [
    "clients/dataforth/scripts/monitor-nas.ps1"
  ],
  "commit_hash": "a3f5b92c...",
  "repository": "claudetools-client-dataforth"
}
```

### Backup References

Track backups in database:

```sql
CREATE TABLE backup_log (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    backup_type VARCHAR(50) CHECK(backup_type IN ('daily', 'manual', 'pre-migration', 'weekly', 'monthly')),
    file_path VARCHAR(500) NOT NULL,
    file_size_bytes BIGINT NOT NULL,
    backup_started_at TIMESTAMP NOT NULL,
    backup_completed_at TIMESTAMP NOT NULL,
    verification_status VARCHAR(50) CHECK(verification_status IN ('passed', 'failed', 'not_verified')),
    verification_details TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,

    INDEX idx_backup_type (backup_type),
    INDEX idx_backup_date (backup_completed_at)
);
```

## Orchestrator Workflow

### During Work Session

1. **Session starts** → Detect mode, client/project
2. **Create files** → In appropriate directory structure
3. **Update database** → Record task context, file paths
4. **Complete tasks** → Track what files were modified
5. **Periodic commits** → Gitea Agent commits changes
6. **Session ends** → Final commit, session log saved

### Example Flow

```python
# User: "Fix the WINS service on D2TESTNAS"

# Orchestrator:
# 1. Determine context
mode = "MSP"
client = "Dataforth"
base_path = "D:/ClaudeTools/clients/dataforth/"

# 2. Launch agents to do work
# ... Coding Agent fixes config file ...

# 3. Files created/modified
files = [
    "configs/nas/smb.conf.overrides",
    "docs/TROUBLESHOOTING.md"
]

# 4. Update database via Database Agent
Database_Agent.update_task(
    task_id=task_id,
    status="completed",
    task_context={
        "files_modified": files,
        "solution": "Fixed syntax error in smb.conf.overrides",
        "verification": "WINS service restarted successfully"
    }
)

# 5. Commit to Gitea via Gitea Agent
Gitea_Agent.commit(
    repository="claudetools-client-dataforth",
    files=files,
    message=generate_commit_message(task_context),
    base_path=base_path
)

# 6. Create session log
session_log_path = f"{base_path}session-logs/2026-01-15-wins-fix.md"
create_session_log(session_log_path, session_data)

# 7. Commit session log
Gitea_Agent.commit(
    repository="claudetools-client-dataforth",
    files=["session-logs/2026-01-15-wins-fix.md"],
    message="Session log: WINS service fix"
)
```

## Benefits

### For User
- **Organized** - Files logically grouped by client/project
- **Version controlled** - Complete history in Gitea
- **Backed up** - Database and files protected
- **Searchable** - Both database queries and file searches
- **Recoverable** - Multiple backup layers

### For System
- **Hybrid storage** - Right tool for right data
- **Context preservation** - Database + files = complete picture
- **Audit trail** - Git history + database records
- **Disaster recovery** - Multiple backup strategies
- **Scalable** - Clean separation of concerns

---

## Summary

**Hybrid Approach:**
- Database → Metadata, relationships, context, queries
- Filesystem → Code, docs, configs (version controlled)
- Gitea → Backup, history, collaboration
- Local backups → Safety net for database

**Organization:**
- `clients/` → MSP Mode work
- `projects/` → Development Mode work
- `normal/` → Normal Mode work
- `backups/` → Database and file backups

**Automation:**
- Gitea Agent → Periodic commits
- Backup Agent → Daily database backups
- Database Agent → Track file references
- Orchestrator → Coordinate all agents

**Result:** Complete, organized, version-controlled, backed-up system.