Session log: Pavon archive cleanup (25TB freed) and OwnCloud integration (35TB external storage)
This commit is contained in:
252
clients/pavon/pavon-cleanup-guide.md
Normal file
252
clients/pavon/pavon-cleanup-guide.md
Normal file
@@ -0,0 +1,252 @@
|
||||
# Pavon Archive Cleanup Guide
|
||||
|
||||
**Server:** 172.16.1.33 (Pavon Unraid)
|
||||
**Script Location:** `/root/pavon_cleanup.sh`
|
||||
**Expected Recovery:** 25.2TB
|
||||
**Date Created:** April 12, 2026
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This script safely deletes camera footage older than 3 years (before April 2023) from Pavon's archive server.
|
||||
|
||||
**What will be deleted:**
|
||||
- Dec 2022: 2.1TB (14,776 files)
|
||||
- Jan 2023: 7.0TB (62,048 files)
|
||||
- Feb 2023: 8.9TB (46,014 files)
|
||||
- Mar 2023: 7.2TB (61,282 files)
|
||||
- **Total: 25.2TB (184,120 files)**
|
||||
|
||||
**What will be kept:**
|
||||
- May 2023 - Oct 2023: 35.1TB
|
||||
- All data within 3-year retention policy
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
### Step 1: Dry-Run (Preview Only - RECOMMENDED FIRST)
|
||||
|
||||
```bash
|
||||
# SSH to Pavon server
|
||||
ssh root@172.16.1.33
|
||||
|
||||
# Run preview (no files deleted)
|
||||
/root/pavon_cleanup.sh
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
- Shows exactly what will be deleted
|
||||
- Calculates space recovery
|
||||
- Lists sample files from each period
|
||||
- **NO FILES ARE DELETED**
|
||||
|
||||
### Step 2: Review the Preview
|
||||
|
||||
Check the output carefully:
|
||||
- Verify date ranges are correct (Dec 2022 - Mar 2023)
|
||||
- Confirm file counts match audit (184,120 files)
|
||||
- Review sample file paths
|
||||
|
||||
### Step 3: Execute Actual Deletion
|
||||
|
||||
**Option A: Interactive execution**
|
||||
```bash
|
||||
# Edit script to disable dry-run
|
||||
nano /root/pavon_cleanup.sh
|
||||
# Change: DRY_RUN=1 to DRY_RUN=0
|
||||
# Save and exit (Ctrl+X, Y, Enter)
|
||||
|
||||
# Run deletion
|
||||
/root/pavon_cleanup.sh
|
||||
```
|
||||
|
||||
**Option B: One-time execution (no script edit)**
|
||||
```bash
|
||||
# Run with dry-run disabled
|
||||
DRY_RUN=0 /root/pavon_cleanup.sh
|
||||
```
|
||||
|
||||
**Confirmation Required:**
|
||||
- Script will ask you to type `DELETE` to confirm
|
||||
- This prevents accidental execution
|
||||
- **Files are permanently deleted** (no recycle bin on Linux)
|
||||
|
||||
---
|
||||
|
||||
## Phased Deletion (Alternative Approach)
|
||||
|
||||
If you want to delete one month at a time:
|
||||
|
||||
### Delete Dec 2022 Only (2.1TB)
|
||||
```bash
|
||||
# Edit script and change PERIODS array to:
|
||||
PERIODS=(
|
||||
"202212:Dec 2022"
|
||||
)
|
||||
```
|
||||
|
||||
### Delete Jan 2023 Only (7.0TB)
|
||||
```bash
|
||||
PERIODS=(
|
||||
"202301:Jan 2023"
|
||||
)
|
||||
```
|
||||
|
||||
### Delete Feb 2023 Only (8.9TB)
|
||||
```bash
|
||||
PERIODS=(
|
||||
"202302:Feb 2023"
|
||||
)
|
||||
```
|
||||
|
||||
### Delete Mar 2023 Only (7.2TB)
|
||||
```bash
|
||||
PERIODS=(
|
||||
"202303:Mar 2023"
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring Progress
|
||||
|
||||
The script provides:
|
||||
- **Real-time output**: Shows each file being deleted
|
||||
- **Progress indicators**: Updates every 1000 files
|
||||
- **Detailed logging**: All actions logged to `/root/cleanup_logs/`
|
||||
|
||||
**To monitor:**
|
||||
```bash
|
||||
# Watch log file in real-time (in another SSH session)
|
||||
tail -f /root/cleanup_logs/cleanup_*.log
|
||||
|
||||
# Check current disk usage
|
||||
df -h /mnt/user
|
||||
|
||||
# Count remaining files
|
||||
find /mnt/user/Storage/cam* -name "Event2022*.avi" | wc -l
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Expected Timeline
|
||||
|
||||
**Deletion speed:** ~500-1000 files/minute (depends on disk I/O)
|
||||
|
||||
| Period | Files | Est. Time |
|
||||
|--------|-------|-----------|
|
||||
| Dec 2022 | 14,776 | 15-30 min |
|
||||
| Jan 2023 | 62,048 | 1-2 hours |
|
||||
| Feb 2023 | 46,014 | 45-90 min |
|
||||
| Mar 2023 | 61,282 | 1-2 hours |
|
||||
| **Total** | **184,120** | **3-5 hours** |
|
||||
|
||||
---
|
||||
|
||||
## Safety Features
|
||||
|
||||
1. **Dry-run default:** Script runs in preview mode unless explicitly changed
|
||||
2. **Confirmation required:** Must type `DELETE` to proceed
|
||||
3. **Detailed logging:** All actions logged to `/root/cleanup_logs/`
|
||||
4. **Pattern-based deletion:** Only deletes files matching `Event2022*.avi` and `Event2023[01-03]*.avi`
|
||||
5. **No recursive wildcards:** Won't accidentally delete wrong directories
|
||||
|
||||
---
|
||||
|
||||
## Verification After Deletion
|
||||
|
||||
```bash
|
||||
# Check new disk usage
|
||||
df -h /mnt/user
|
||||
|
||||
# Verify old files are gone
|
||||
find /mnt/user/Storage/cam* -name "Event2022*.avi" | wc -l # Should be 0
|
||||
find /mnt/user/Storage/cam* -name "Event202301*.avi" | wc -l # Should be 0
|
||||
find /mnt/user/Storage/cam* -name "Event202302*.avi" | wc -l # Should be 0
|
||||
find /mnt/user/Storage/cam* -name "Event202303*.avi" | wc -l # Should be 0
|
||||
|
||||
# Verify remaining files intact
|
||||
find /mnt/user/Storage/cam* -name "Event202305*.avi" | wc -l # Should have files
|
||||
find /mnt/user/Storage/cam* -name "Event202306*.avi" | wc -l # Should have files
|
||||
|
||||
# Check logs
|
||||
ls -lh /root/cleanup_logs/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
**Important:** Once deleted, files cannot be recovered unless you have backups.
|
||||
|
||||
**Before deletion:**
|
||||
- If unsure, create backup: `rsync -av /mnt/user/Storage /mnt/user/Backups/pavon_archive_backup/`
|
||||
- Takes ~6-8 hours to backup 60TB, requires 60TB free space
|
||||
|
||||
**No backups exist on Jupiter for this data** (confirmed during audit).
|
||||
|
||||
---
|
||||
|
||||
## Post-Cleanup Actions
|
||||
|
||||
After successful deletion:
|
||||
|
||||
1. **Verify space recovery:**
|
||||
```bash
|
||||
df -h /mnt/user
|
||||
# Should show ~84TB free (was 59TB)
|
||||
```
|
||||
|
||||
2. **Set up automated retention (optional):**
|
||||
- Create monthly cron job
|
||||
- Auto-delete data >3 years old
|
||||
- Email notifications
|
||||
|
||||
3. **Document in credentials.md:**
|
||||
- Update Pavon server notes
|
||||
- Record cleanup date
|
||||
- Note new available capacity
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Script hangs or runs slowly
|
||||
- Normal for large deletions (184K files)
|
||||
- Check progress: `tail -f /root/cleanup_logs/cleanup_*.log`
|
||||
- Monitor disk I/O: `iotop` (if installed)
|
||||
|
||||
### "Permission denied" errors
|
||||
- Run as root: `sudo /root/pavon_cleanup.sh`
|
||||
- Check file ownership: `ls -l /mnt/user/Storage/cam*/`
|
||||
|
||||
### Want to cancel during execution
|
||||
- Press `Ctrl+C` to stop
|
||||
- Files deleted so far are gone
|
||||
- Remaining files are safe
|
||||
- Can resume by running script again (only deletes what remains)
|
||||
|
||||
### Disk space not showing as free
|
||||
- Unraid may need array refresh
|
||||
- Wait 5-10 minutes for system to update
|
||||
- Run: `du -sh /mnt/user/Storage` to verify actual usage
|
||||
|
||||
---
|
||||
|
||||
## Support
|
||||
|
||||
**Script location on local machine:**
|
||||
`/Users/azcomputerguru/ClaudeTools/temp/pavon_cleanup.sh`
|
||||
|
||||
**Logs location:**
|
||||
`/root/cleanup_logs/` on Pavon server
|
||||
|
||||
**Contact:** Check session logs for questions
|
||||
|
||||
---
|
||||
|
||||
**Created:** April 12, 2026
|
||||
**Audit performed:** April 12, 2026
|
||||
**Last updated:** April 12, 2026
|
||||
Reference in New Issue
Block a user