claudetools/clients/internal-infrastructure/ix-server-issues-2026-01-13.md

# IX Server Critical Performance Issues

**Server:** ix.azcomputerguru.com (172.16.3.10 / 72.194.62.5)
**Report Date:** 2026-01-13
**Status:** Documented - Action Required
**Priority:** CRITICAL

## Executive Summary

Comprehensive scan of ix.azcomputerguru.com web hosting server revealed critical performance issues across multiple client sites. Primary issues: massive error logs (468MB on arizonahatters.com), database bloat (310MB on peacefulspirit.com), and Wordfence-induced memory exhaustion.

---

## Critical Priority Sites

### 1. arizonahatters.com - MOST URGENT

**Error Log:** 468MB
**PHP Memory Errors:** 429 occurrences
**Database:** 24.5MB (Wordfence bloat: wp_wffilemods 8.52MB, wp_wfknownfilelist 4.52MB)

**Issue:** Wordfence file scanning causing continuous memory exhaustion

**Impact:**
- Site performance degraded
- Server resources exhausted
- Risk of complete service failure

**Action Required:**
1. Disable Wordfence file scanning temporarily
2. Clear Wordfence file modification tables
3. Truncate error log: `/home/arizonahatters/public_html/wp-content/debug.log`
4. Re-enable Wordfence with adjusted settings (scan schedule, memory limit)

**Commands:**
```bash
# Backup then truncate error log
ssh root@172.16.3.10
cd /home/arizonahatters/public_html/wp-content/
cp debug.log debug.log.backup.2026-01-13
> debug.log

# Database cleanup (via WP-CLI)
wp db query "TRUNCATE TABLE wp_wffilemods;" --path=/home/arizonahatters/public_html/
wp db query "TRUNCATE TABLE wp_wfknownfilelist;" --path=/home/arizonahatters/public_html/
```

---

### 2. peacefulspirit.com

**Error Log:** 4.0MB
**PHP Memory Errors:** 2 occurrences
**Database:** 310MB! (wp_wpml_mails: 156MB, wp_gf_entry_meta: 96MB)

**Issue:** WPML email logs and Gravity Forms data bloat

**Impact:**
- Slow database queries
- Backup size excessive
- Disk space waste

**Action Required:**
1. Truncate WPML email logs table
2. Archive or delete old Gravity Forms entries
3. Configure WPML to limit email log retention
4. Implement Gravity Forms entry retention policy

**Commands:**
```bash
# WPML email logs cleanup
wp db query "TRUNCATE TABLE wp_wpml_mails;" --path=/home/peacefulspirit/public_html/

# Gravity Forms cleanup (entries older than 1 year)
wp db query "DELETE FROM wp_gf_entry WHERE date_created < DATE_SUB(NOW(), INTERVAL 1 YEAR);" --path=/home/peacefulspirit/public_html/
wp db query "DELETE FROM wp_gf_entry_meta WHERE entry_id NOT IN (SELECT id FROM wp_gf_entry);" --path=/home/peacefulspirit/public_html/
```

---

## High Priority Sites (>50MB Error Logs)

| Site | Error Log Size | Primary Issue |
|------|---------------|---------------|
| desertfox.com | 215MB | Unknown - needs investigation |
| outaboundssports.com | 208MB | Unknown - needs investigation |
| rrspc.com | 183MB | Unknown - needs investigation |
| farwest.com | 100MB | Unknown - needs investigation |
| fsgtucson.com | 64MB | Unknown - needs investigation |
| tonystech.com | 54MB | Unknown - needs investigation |
| phxpropane.com | 52MB | Unknown - needs investigation |
| rednourlaw.com | 50MB | Unknown - needs investigation |
| gurushow.com | 40MB | Unknown - needs investigation |
| cryoweave.com | 37MB | Unknown - needs investigation |
| bruceext.com | 31MB | Unknown - needs investigation |

**Recommended Action:**
1. Rotate error logs (backup and truncate)
2. Analyze recent errors for patterns
3. Address root causes (plugin conflicts, PHP errors, etc.)
4. Implement log rotation via logrotate

---

## Medium Priority Sites (Debug Logs)

| Site | Debug Log Size | Additional Issues |
|------|---------------|-------------------|
| gentlemansacres.com | debug.log: 350MB | N/A |
| azrestaurant.com | debug.log: 181MB, itsec_logs: 53MB | iThemes Security logs |
| rsi.com | debug.log: 166MB | N/A |
| voicesofthewest.com | akeeba log: 106MB | Backup log bloat |

**Action Required:**
- Disable WP_DEBUG in production (wp-config.php)
- Truncate debug logs
- Configure iThemes Security log retention
- Clean up Akeeba backup logs

---

## Common Issues Found

### 1. Wordfence Database Bloat (Most Sites)

**Tables:**
- wp_wffilemods: 1.4-8.52MB
- wp_wfknownfilelist: 0.86-4.52MB
- wp_wfconfig: Up to 3.30MB

**Solution:**
```sql
-- Run on each affected site
TRUNCATE TABLE wp_wffilemods;
TRUNCATE TABLE wp_wfknownfilelist;
DELETE FROM wp_wfconfig WHERE name LIKE '%filemod%';
```

### 2. Email/Form Logs

**Common Culprits:**
- WPML email logs (wp_wpml_mails)
- Gravity Forms entries (wp_gf_entry, wp_gf_entry_meta)
- Post SMTP logs
- Action Scheduler logs

**Solution:** Implement retention policies, truncate old data

### 3. Old Backups (Disk Space)

| Site | Backup Size | Age |
|------|-------------|-----|
| acepickupparts | 1.6GB | Various |
| azcomputerguru | 3GB+ | Various |
| sundanzer | 2GB | Various |
| berman | 388MB | 2019 |
| rrspc | 314MB | 2021 |

**Action Required:** Archive to offsite storage, delete from web server

---

## Scan Commands

### Full Site Scan
```bash
ssh root@172.16.3.10
/root/scan_sites.sh
cat /root/site_scan_report.txt
```

### Database Bloat Check
```bash
ssh root@172.16.3.10
/root/check_dbs.sh
cat /root/db_bloat_report.txt
```

### View Critical Issues
```bash
ssh root@172.16.3.10
cat /root/URGENT_SITE_ISSUES.txt
```

---

## Automation Recommendations

### 1. Log Rotation

**Create:** `/etc/logrotate.d/wordpress-error-logs`
```
/home/*/public_html/wp-content/debug.log {
    weekly
    rotate 4
    compress
    delaycompress
    missingok
    notifempty
    create 644 root root
}
```

### 2. Database Maintenance Script

**Create:** `/root/wordpress-db-maintenance.sh`
```bash
#!/bin/bash
# WordPress database maintenance - run weekly

for site in /home/*/public_html; do
    if [ -f "$site/wp-config.php" ]; then
        echo "Cleaning $site..."

        # Wordfence cleanup
        wp db query "TRUNCATE TABLE wp_wffilemods;" --path="$site" 2>/dev/null
        wp db query "TRUNCATE TABLE wp_wfknownfilelist;" --path="$site" 2>/dev/null

        # Optimize all tables
        wp db optimize --path="$site" 2>/dev/null
    fi
done
```

### 3. Monitoring Alerts

**Create:** `/root/monitor-disk-usage.sh`
```bash
#!/bin/bash
# Alert if any site error log >100MB

find /home/*/public_html/wp-content/ -name "debug.log" -size +100M -exec ls -lh {} \; | \
  mail -s "IX Server: Large error logs detected" mike@azcomputerguru.com
```

---

## Server Resources

### Current Usage
```bash
# Check disk space
df -h /home

# Check memory usage
free -h

# Check CPU load
uptime
```

### Optimization Recommendations

1. **OPcache:** Ensure enabled and properly configured
2. **MariaDB:** Tune query cache and buffer pool size
3. **PHP-FPM:** Adjust pm.max_children based on memory
4. **Apache/LiteSpeed:** Enable HTTP/2, optimize workers

---

## Client Communication Template

**Subject:** Website Performance Maintenance Required

**Body:**
```
Hello [Client Name],

During our routine server maintenance, we identified some performance
issues affecting your website that require attention:

1. Error logs have grown to [SIZE], indicating [ISSUE]
2. Database optimization needed due to [BLOAT TYPE]

Recommended Actions:
- [SPECIFIC ACTION 1]
- [SPECIFIC ACTION 2]

Impact: [EXPECTED DOWNTIME/IMPROVEMENT]

We can schedule this work at your convenience. Please let us know
your preferred maintenance window.

Best regards,
Arizona Computer Guru Support
```

---

## Follow-Up Tasks

- [ ] Contact each critical priority client
- [ ] Schedule maintenance windows
- [ ] Execute cleanup on arizonahatters.com
- [ ] Execute cleanup on peacefulspirit.com
- [ ] Implement log rotation across all sites
- [ ] Create database maintenance cron job
- [ ] Set up monitoring alerts
- [ ] Document lessons learned
- [ ] Review Wordfence configuration across all sites
- [ ] Audit backup retention policies

---

## Server Access

```bash
# External SSH
ssh root@ix.azcomputerguru.com

# Internal SSH
ssh root@172.16.3.10

# WHM
https://ix.azcomputerguru.com:2087

# cPanel (example)
https://ix.azcomputerguru.com:2083
```

---

## Related Documentation

**Original Report:** `~/claude-projects/IX_SERVER_CRITICAL_ISSUES_2026-01-13.md`

**Session Logs:**
- Various client work sessions documented in `~/claude-projects/session-logs/`

**Scripts Location:**
- `/root/scan_sites.sh`
- `/root/check_dbs.sh`
- `/root/URGENT_SITE_ISSUES.txt`

---

## Project History

**2026-01-13:** Initial comprehensive server scan and issue documentation
**2026-01-22:** Imported to ClaudeTools project tracking system

---

**Status:** Documented - Awaiting Action
**Owner:** Arizona Computer Guru Operations Team
**Next Review:** After critical issues resolved