feat: Major directory reorganization and cleanup

Reorganized project structure for better maintainability and reduced
disk usage by 95.9% (11 GB -> 451 MB).

Directory Reorganization (85% reduction in root files):
- Created docs/ with subdirectories (deployment, testing, database, etc.)
- Created infrastructure/vpn-configs/ for VPN scripts
- Moved 90+ files from root to organized locations
- Archived obsolete documentation (context system, offline mode, zombie debugging)
- Moved all test files to tests/ directory
- Root directory: 119 files -> 18 files

Disk Cleanup (10.55 GB recovered):
- Deleted Rust build artifacts: 9.6 GB (target/ directories)
- Deleted Python virtual environments: 161 MB (venv/ directories)
- Deleted Python cache: 50 KB (__pycache__/)

New Structure:
- docs/ - All documentation organized by category
- docs/archives/ - Obsolete but preserved documentation
- infrastructure/ - VPN configs and SSH setup
- tests/ - All test files consolidated
- logs/ - Ready for future logs

Benefits:
- Cleaner root directory (18 vs 119 files)
- Logical organization of documentation
- 95.9% disk space reduction
- Faster navigation and discovery
- Better portability (build artifacts excluded)

Build artifacts can be regenerated:
- Rust: cargo build --release (5-15 min per project)
- Python: pip install -r requirements.txt (2-3 min)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
2026-01-18 20:42:28 -07:00
parent 89e5118306
commit 06f7617718
96 changed files with 54 additions and 2639 deletions

View File

@@ -0,0 +1,221 @@
# Credential Scanner Quick Reference
**Module:** `api/utils/credential_scanner`
**Purpose:** Import credentials from files to database with auto-encryption
---
## Quick Start
```python
from api.database import SessionLocal
from api.utils.credential_scanner import scan_and_import_credentials
db = SessionLocal()
try:
results = scan_and_import_credentials(
base_path="C:/Projects/MyClient",
db=db,
client_id="uuid-here" # Optional
)
print(f"Imported: {results['credentials_imported']}")
finally:
db.close()
```
---
## Functions
### 1. `scan_for_credential_files(base_path)`
Find all credential files in directory tree.
**Returns:** `List[str]` - File paths
**Finds:**
- credentials.md, credentials.txt
- passwords.md, passwords.txt
- .env, .env.local, .env.production
- secrets.md, auth.md
---
### 2. `parse_credential_file(file_path)`
Parse credentials from a file.
**Returns:** `List[Dict]` - Credential dictionaries
**Example output:**
```python
[
{
"service_name": "Gitea Admin",
"credential_type": "password",
"username": "admin",
"password": "SecurePass123!"
},
...
]
```
---
### 3. `import_credentials_to_db(db, credentials, client_id=None, user_id="system_import")`
Import credentials with auto-encryption.
**Returns:** `int` - Count of imported credentials
**Features:**
- Auto-encrypts sensitive fields (AES-256-GCM)
- Creates audit log entries
- Never logs plaintext values
- Continues on errors
---
### 4. `scan_and_import_credentials(base_path, db, client_id=None, user_id="system_import")`
Complete workflow in one call.
**Returns:** `Dict[str, int]`
```python
{
"files_found": 3,
"credentials_parsed": 8,
"credentials_imported": 8
}
```
---
## File Formats
### Markdown (.md)
```markdown
## Service Name
Username: admin
Password: secret123
API Key: sk-1234567890
URL: https://example.com
Notes: Additional info
```
### Environment (.env)
```bash
DATABASE_URL=mysql://user:pass@host/db
API_KEY=sk-1234567890
SECRET_TOKEN=abc123def456
```
### Text (.txt)
Same as Markdown format
---
## Credential Types Auto-Detected
| Value Pattern | Type | Field |
|--------------|------|-------|
| `sk-*` | api_key | api_key |
| `ghp_*` | api_key | api_key |
| `mysql://...` | connection_string | connection_string |
| `-----BEGIN...` | ssh_key | password |
| JWT (3 parts) | jwt | token |
| Default | password | password |
---
## Security
**Encryption:** AES-256-GCM via `credential_service`
**Audit:** Every import logged to `credential_audit_log`
**Logging:** Never logs plaintext credentials
---
## Command Line
```bash
# Preview
python example_credential_import.py /path --preview
# Import
python example_credential_import.py /path --client-id "uuid"
```
---
## Common Workflows
### Import from Client Directory
```python
db = SessionLocal()
try:
results = scan_and_import_credentials(
"C:/Projects/ClientA",
db,
client_id="client-uuid"
)
finally:
db.close()
```
### Preview Before Import
```python
files = scan_for_credential_files("/path")
for f in files:
creds = parse_credential_file(f)
print(f"{f}: {len(creds)} credentials")
```
### Import with Error Handling
```python
files = scan_for_credential_files("/path")
for file_path in files:
try:
creds = parse_credential_file(file_path)
count = import_credentials_to_db(db, creds)
print(f"{count} from {file_path}")
except Exception as e:
print(f"✗ Failed: {e}")
```
---
## Testing
```bash
python test_credential_scanner.py
# All 5 tests should pass
```
---
## Documentation
- **Full Guide:** `CREDENTIAL_SCANNER_GUIDE.md`
- **Summary:** `CREDENTIAL_SCANNER_SUMMARY.md`
- **Examples:** `example_credential_import.py`
- **Tests:** `test_credential_scanner.py`
---
## Troubleshooting
**No files found?**
- Check base_path exists
- Verify file names match patterns
- Ensure not in excluded dirs (.git, node_modules)
**Parsing errors?**
- Verify file format (headers, key:value pairs)
- Check UTF-8 encoding
- Ensure recognized key names
**Import fails?**
- Check database connection
- Verify ENCRYPTION_KEY set
- Check client_id exists (if provided)
---
**Quick Help:** See `CREDENTIAL_SCANNER_GUIDE.md` for complete documentation