Reorganized project structure for better maintainability and reduced disk usage by 95.9% (11 GB -> 451 MB). Directory Reorganization (85% reduction in root files): - Created docs/ with subdirectories (deployment, testing, database, etc.) - Created infrastructure/vpn-configs/ for VPN scripts - Moved 90+ files from root to organized locations - Archived obsolete documentation (context system, offline mode, zombie debugging) - Moved all test files to tests/ directory - Root directory: 119 files -> 18 files Disk Cleanup (10.55 GB recovered): - Deleted Rust build artifacts: 9.6 GB (target/ directories) - Deleted Python virtual environments: 161 MB (venv/ directories) - Deleted Python cache: 50 KB (__pycache__/) New Structure: - docs/ - All documentation organized by category - docs/archives/ - Obsolete but preserved documentation - infrastructure/ - VPN configs and SSH setup - tests/ - All test files consolidated - logs/ - Ready for future logs Benefits: - Cleaner root directory (18 vs 119 files) - Logical organization of documentation - 95.9% disk space reduction - Faster navigation and discovery - Better portability (build artifacts excluded) Build artifacts can be regenerated: - Rust: cargo build --release (5-15 min per project) - Python: pip install -r requirements.txt (2-3 min) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
222 lines
4.2 KiB
Markdown
222 lines
4.2 KiB
Markdown
# Credential Scanner Quick Reference
|
|
|
|
**Module:** `api/utils/credential_scanner`
|
|
**Purpose:** Import credentials from files to database with auto-encryption
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
```python
|
|
from api.database import SessionLocal
|
|
from api.utils.credential_scanner import scan_and_import_credentials
|
|
|
|
db = SessionLocal()
|
|
try:
|
|
results = scan_and_import_credentials(
|
|
base_path="C:/Projects/MyClient",
|
|
db=db,
|
|
client_id="uuid-here" # Optional
|
|
)
|
|
print(f"Imported: {results['credentials_imported']}")
|
|
finally:
|
|
db.close()
|
|
```
|
|
|
|
---
|
|
|
|
## Functions
|
|
|
|
### 1. `scan_for_credential_files(base_path)`
|
|
Find all credential files in directory tree.
|
|
|
|
**Returns:** `List[str]` - File paths
|
|
|
|
**Finds:**
|
|
- credentials.md, credentials.txt
|
|
- passwords.md, passwords.txt
|
|
- .env, .env.local, .env.production
|
|
- secrets.md, auth.md
|
|
|
|
---
|
|
|
|
### 2. `parse_credential_file(file_path)`
|
|
Parse credentials from a file.
|
|
|
|
**Returns:** `List[Dict]` - Credential dictionaries
|
|
|
|
**Example output:**
|
|
```python
|
|
[
|
|
{
|
|
"service_name": "Gitea Admin",
|
|
"credential_type": "password",
|
|
"username": "admin",
|
|
"password": "SecurePass123!"
|
|
},
|
|
...
|
|
]
|
|
```
|
|
|
|
---
|
|
|
|
### 3. `import_credentials_to_db(db, credentials, client_id=None, user_id="system_import")`
|
|
Import credentials with auto-encryption.
|
|
|
|
**Returns:** `int` - Count of imported credentials
|
|
|
|
**Features:**
|
|
- Auto-encrypts sensitive fields (AES-256-GCM)
|
|
- Creates audit log entries
|
|
- Never logs plaintext values
|
|
- Continues on errors
|
|
|
|
---
|
|
|
|
### 4. `scan_and_import_credentials(base_path, db, client_id=None, user_id="system_import")`
|
|
Complete workflow in one call.
|
|
|
|
**Returns:** `Dict[str, int]`
|
|
```python
|
|
{
|
|
"files_found": 3,
|
|
"credentials_parsed": 8,
|
|
"credentials_imported": 8
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## File Formats
|
|
|
|
### Markdown (.md)
|
|
```markdown
|
|
## Service Name
|
|
Username: admin
|
|
Password: secret123
|
|
API Key: sk-1234567890
|
|
URL: https://example.com
|
|
Notes: Additional info
|
|
```
|
|
|
|
### Environment (.env)
|
|
```bash
|
|
DATABASE_URL=mysql://user:pass@host/db
|
|
API_KEY=sk-1234567890
|
|
SECRET_TOKEN=abc123def456
|
|
```
|
|
|
|
### Text (.txt)
|
|
Same as Markdown format
|
|
|
|
---
|
|
|
|
## Credential Types Auto-Detected
|
|
|
|
| Value Pattern | Type | Field |
|
|
|--------------|------|-------|
|
|
| `sk-*` | api_key | api_key |
|
|
| `ghp_*` | api_key | api_key |
|
|
| `mysql://...` | connection_string | connection_string |
|
|
| `-----BEGIN...` | ssh_key | password |
|
|
| JWT (3 parts) | jwt | token |
|
|
| Default | password | password |
|
|
|
|
---
|
|
|
|
## Security
|
|
|
|
**Encryption:** AES-256-GCM via `credential_service`
|
|
**Audit:** Every import logged to `credential_audit_log`
|
|
**Logging:** Never logs plaintext credentials
|
|
|
|
---
|
|
|
|
## Command Line
|
|
|
|
```bash
|
|
# Preview
|
|
python example_credential_import.py /path --preview
|
|
|
|
# Import
|
|
python example_credential_import.py /path --client-id "uuid"
|
|
```
|
|
|
|
---
|
|
|
|
## Common Workflows
|
|
|
|
### Import from Client Directory
|
|
```python
|
|
db = SessionLocal()
|
|
try:
|
|
results = scan_and_import_credentials(
|
|
"C:/Projects/ClientA",
|
|
db,
|
|
client_id="client-uuid"
|
|
)
|
|
finally:
|
|
db.close()
|
|
```
|
|
|
|
### Preview Before Import
|
|
```python
|
|
files = scan_for_credential_files("/path")
|
|
for f in files:
|
|
creds = parse_credential_file(f)
|
|
print(f"{f}: {len(creds)} credentials")
|
|
```
|
|
|
|
### Import with Error Handling
|
|
```python
|
|
files = scan_for_credential_files("/path")
|
|
for file_path in files:
|
|
try:
|
|
creds = parse_credential_file(file_path)
|
|
count = import_credentials_to_db(db, creds)
|
|
print(f"✓ {count} from {file_path}")
|
|
except Exception as e:
|
|
print(f"✗ Failed: {e}")
|
|
```
|
|
|
|
---
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
python test_credential_scanner.py
|
|
# All 5 tests should pass
|
|
```
|
|
|
|
---
|
|
|
|
## Documentation
|
|
|
|
- **Full Guide:** `CREDENTIAL_SCANNER_GUIDE.md`
|
|
- **Summary:** `CREDENTIAL_SCANNER_SUMMARY.md`
|
|
- **Examples:** `example_credential_import.py`
|
|
- **Tests:** `test_credential_scanner.py`
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
**No files found?**
|
|
- Check base_path exists
|
|
- Verify file names match patterns
|
|
- Ensure not in excluded dirs (.git, node_modules)
|
|
|
|
**Parsing errors?**
|
|
- Verify file format (headers, key:value pairs)
|
|
- Check UTF-8 encoding
|
|
- Ensure recognized key names
|
|
|
|
**Import fails?**
|
|
- Check database connection
|
|
- Verify ENCRYPTION_KEY set
|
|
- Check client_id exists (if provided)
|
|
|
|
---
|
|
|
|
**Quick Help:** See `CREDENTIAL_SCANNER_GUIDE.md` for complete documentation
|