docs: Add comprehensive project documentation from claude-projects scan
Added: - PROJECTS_INDEX.md - Master catalog of 7 active projects - GURURMM_API_ACCESS.md - Complete API documentation and credentials - clients/dataforth/dos-test-machines/README.md - DOS update system docs - clients/grabb-durando/website-migration/README.md - Migration procedures - clients/internal-infrastructure/ix-server-issues-2026-01-13.md - Server issues - projects/msp-tools/guru-connect/README.md - Remote desktop architecture - projects/msp-tools/toolkit/README.md - MSP PowerShell tools - projects/internal/acg-website-2025/README.md - Website rebuild docs - test_gururmm_api.py - GuruRMM API testing script Modified: - credentials.md - Added GuruRMM database and API credentials - GuruRMM agent integration files (WebSocket transport) Total: 38,000+ words of comprehensive project documentation Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
346
clients/internal-infrastructure/ix-server-issues-2026-01-13.md
Normal file
346
clients/internal-infrastructure/ix-server-issues-2026-01-13.md
Normal file
@@ -0,0 +1,346 @@
|
||||
# IX Server Critical Performance Issues
|
||||
|
||||
**Server:** ix.azcomputerguru.com (172.16.3.10 / 72.194.62.5)
|
||||
**Report Date:** 2026-01-13
|
||||
**Status:** Documented - Action Required
|
||||
**Priority:** CRITICAL
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Comprehensive scan of ix.azcomputerguru.com web hosting server revealed critical performance issues across multiple client sites. Primary issues: massive error logs (468MB on arizonahatters.com), database bloat (310MB on peacefulspirit.com), and Wordfence-induced memory exhaustion.
|
||||
|
||||
---
|
||||
|
||||
## Critical Priority Sites
|
||||
|
||||
### 1. arizonahatters.com - MOST URGENT
|
||||
|
||||
**Error Log:** 468MB
|
||||
**PHP Memory Errors:** 429 occurrences
|
||||
**Database:** 24.5MB (Wordfence bloat: wp_wffilemods 8.52MB, wp_wfknownfilelist 4.52MB)
|
||||
|
||||
**Issue:** Wordfence file scanning causing continuous memory exhaustion
|
||||
|
||||
**Impact:**
|
||||
- Site performance degraded
|
||||
- Server resources exhausted
|
||||
- Risk of complete service failure
|
||||
|
||||
**Action Required:**
|
||||
1. Disable Wordfence file scanning temporarily
|
||||
2. Clear Wordfence file modification tables
|
||||
3. Truncate error log: `/home/arizonahatters/public_html/wp-content/debug.log`
|
||||
4. Re-enable Wordfence with adjusted settings (scan schedule, memory limit)
|
||||
|
||||
**Commands:**
|
||||
```bash
|
||||
# Backup then truncate error log
|
||||
ssh root@172.16.3.10
|
||||
cd /home/arizonahatters/public_html/wp-content/
|
||||
cp debug.log debug.log.backup.2026-01-13
|
||||
> debug.log
|
||||
|
||||
# Database cleanup (via WP-CLI)
|
||||
wp db query "TRUNCATE TABLE wp_wffilemods;" --path=/home/arizonahatters/public_html/
|
||||
wp db query "TRUNCATE TABLE wp_wfknownfilelist;" --path=/home/arizonahatters/public_html/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. peacefulspirit.com
|
||||
|
||||
**Error Log:** 4.0MB
|
||||
**PHP Memory Errors:** 2 occurrences
|
||||
**Database:** 310MB! (wp_wpml_mails: 156MB, wp_gf_entry_meta: 96MB)
|
||||
|
||||
**Issue:** WPML email logs and Gravity Forms data bloat
|
||||
|
||||
**Impact:**
|
||||
- Slow database queries
|
||||
- Backup size excessive
|
||||
- Disk space waste
|
||||
|
||||
**Action Required:**
|
||||
1. Truncate WPML email logs table
|
||||
2. Archive or delete old Gravity Forms entries
|
||||
3. Configure WPML to limit email log retention
|
||||
4. Implement Gravity Forms entry retention policy
|
||||
|
||||
**Commands:**
|
||||
```bash
|
||||
# WPML email logs cleanup
|
||||
wp db query "TRUNCATE TABLE wp_wpml_mails;" --path=/home/peacefulspirit/public_html/
|
||||
|
||||
# Gravity Forms cleanup (entries older than 1 year)
|
||||
wp db query "DELETE FROM wp_gf_entry WHERE date_created < DATE_SUB(NOW(), INTERVAL 1 YEAR);" --path=/home/peacefulspirit/public_html/
|
||||
wp db query "DELETE FROM wp_gf_entry_meta WHERE entry_id NOT IN (SELECT id FROM wp_gf_entry);" --path=/home/peacefulspirit/public_html/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## High Priority Sites (>50MB Error Logs)
|
||||
|
||||
| Site | Error Log Size | Primary Issue |
|
||||
|------|---------------|---------------|
|
||||
| desertfox.com | 215MB | Unknown - needs investigation |
|
||||
| outaboundssports.com | 208MB | Unknown - needs investigation |
|
||||
| rrspc.com | 183MB | Unknown - needs investigation |
|
||||
| farwest.com | 100MB | Unknown - needs investigation |
|
||||
| fsgtucson.com | 64MB | Unknown - needs investigation |
|
||||
| tonystech.com | 54MB | Unknown - needs investigation |
|
||||
| phxpropane.com | 52MB | Unknown - needs investigation |
|
||||
| rednourlaw.com | 50MB | Unknown - needs investigation |
|
||||
| gurushow.com | 40MB | Unknown - needs investigation |
|
||||
| cryoweave.com | 37MB | Unknown - needs investigation |
|
||||
| bruceext.com | 31MB | Unknown - needs investigation |
|
||||
|
||||
**Recommended Action:**
|
||||
1. Rotate error logs (backup and truncate)
|
||||
2. Analyze recent errors for patterns
|
||||
3. Address root causes (plugin conflicts, PHP errors, etc.)
|
||||
4. Implement log rotation via logrotate
|
||||
|
||||
---
|
||||
|
||||
## Medium Priority Sites (Debug Logs)
|
||||
|
||||
| Site | Debug Log Size | Additional Issues |
|
||||
|------|---------------|-------------------|
|
||||
| gentlemansacres.com | debug.log: 350MB | N/A |
|
||||
| azrestaurant.com | debug.log: 181MB, itsec_logs: 53MB | iThemes Security logs |
|
||||
| rsi.com | debug.log: 166MB | N/A |
|
||||
| voicesofthewest.com | akeeba log: 106MB | Backup log bloat |
|
||||
|
||||
**Action Required:**
|
||||
- Disable WP_DEBUG in production (wp-config.php)
|
||||
- Truncate debug logs
|
||||
- Configure iThemes Security log retention
|
||||
- Clean up Akeeba backup logs
|
||||
|
||||
---
|
||||
|
||||
## Common Issues Found
|
||||
|
||||
### 1. Wordfence Database Bloat (Most Sites)
|
||||
|
||||
**Tables:**
|
||||
- wp_wffilemods: 1.4-8.52MB
|
||||
- wp_wfknownfilelist: 0.86-4.52MB
|
||||
- wp_wfconfig: Up to 3.30MB
|
||||
|
||||
**Solution:**
|
||||
```sql
|
||||
-- Run on each affected site
|
||||
TRUNCATE TABLE wp_wffilemods;
|
||||
TRUNCATE TABLE wp_wfknownfilelist;
|
||||
DELETE FROM wp_wfconfig WHERE name LIKE '%filemod%';
|
||||
```
|
||||
|
||||
### 2. Email/Form Logs
|
||||
|
||||
**Common Culprits:**
|
||||
- WPML email logs (wp_wpml_mails)
|
||||
- Gravity Forms entries (wp_gf_entry, wp_gf_entry_meta)
|
||||
- Post SMTP logs
|
||||
- Action Scheduler logs
|
||||
|
||||
**Solution:** Implement retention policies, truncate old data
|
||||
|
||||
### 3. Old Backups (Disk Space)
|
||||
|
||||
| Site | Backup Size | Age |
|
||||
|------|-------------|-----|
|
||||
| acepickupparts | 1.6GB | Various |
|
||||
| azcomputerguru | 3GB+ | Various |
|
||||
| sundanzer | 2GB | Various |
|
||||
| berman | 388MB | 2019 |
|
||||
| rrspc | 314MB | 2021 |
|
||||
|
||||
**Action Required:** Archive to offsite storage, delete from web server
|
||||
|
||||
---
|
||||
|
||||
## Scan Commands
|
||||
|
||||
### Full Site Scan
|
||||
```bash
|
||||
ssh root@172.16.3.10
|
||||
/root/scan_sites.sh
|
||||
cat /root/site_scan_report.txt
|
||||
```
|
||||
|
||||
### Database Bloat Check
|
||||
```bash
|
||||
ssh root@172.16.3.10
|
||||
/root/check_dbs.sh
|
||||
cat /root/db_bloat_report.txt
|
||||
```
|
||||
|
||||
### View Critical Issues
|
||||
```bash
|
||||
ssh root@172.16.3.10
|
||||
cat /root/URGENT_SITE_ISSUES.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Automation Recommendations
|
||||
|
||||
### 1. Log Rotation
|
||||
|
||||
**Create:** `/etc/logrotate.d/wordpress-error-logs`
|
||||
```
|
||||
/home/*/public_html/wp-content/debug.log {
|
||||
weekly
|
||||
rotate 4
|
||||
compress
|
||||
delaycompress
|
||||
missingok
|
||||
notifempty
|
||||
create 644 root root
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Database Maintenance Script
|
||||
|
||||
**Create:** `/root/wordpress-db-maintenance.sh`
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# WordPress database maintenance - run weekly
|
||||
|
||||
for site in /home/*/public_html; do
|
||||
if [ -f "$site/wp-config.php" ]; then
|
||||
echo "Cleaning $site..."
|
||||
|
||||
# Wordfence cleanup
|
||||
wp db query "TRUNCATE TABLE wp_wffilemods;" --path="$site" 2>/dev/null
|
||||
wp db query "TRUNCATE TABLE wp_wfknownfilelist;" --path="$site" 2>/dev/null
|
||||
|
||||
# Optimize all tables
|
||||
wp db optimize --path="$site" 2>/dev/null
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
### 3. Monitoring Alerts
|
||||
|
||||
**Create:** `/root/monitor-disk-usage.sh`
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Alert if any site error log >100MB
|
||||
|
||||
find /home/*/public_html/wp-content/ -name "debug.log" -size +100M -exec ls -lh {} \; | \
|
||||
mail -s "IX Server: Large error logs detected" mike@azcomputerguru.com
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Server Resources
|
||||
|
||||
### Current Usage
|
||||
```bash
|
||||
# Check disk space
|
||||
df -h /home
|
||||
|
||||
# Check memory usage
|
||||
free -h
|
||||
|
||||
# Check CPU load
|
||||
uptime
|
||||
```
|
||||
|
||||
### Optimization Recommendations
|
||||
|
||||
1. **OPcache:** Ensure enabled and properly configured
|
||||
2. **MariaDB:** Tune query cache and buffer pool size
|
||||
3. **PHP-FPM:** Adjust pm.max_children based on memory
|
||||
4. **Apache/LiteSpeed:** Enable HTTP/2, optimize workers
|
||||
|
||||
---
|
||||
|
||||
## Client Communication Template
|
||||
|
||||
**Subject:** Website Performance Maintenance Required
|
||||
|
||||
**Body:**
|
||||
```
|
||||
Hello [Client Name],
|
||||
|
||||
During our routine server maintenance, we identified some performance
|
||||
issues affecting your website that require attention:
|
||||
|
||||
1. Error logs have grown to [SIZE], indicating [ISSUE]
|
||||
2. Database optimization needed due to [BLOAT TYPE]
|
||||
|
||||
Recommended Actions:
|
||||
- [SPECIFIC ACTION 1]
|
||||
- [SPECIFIC ACTION 2]
|
||||
|
||||
Impact: [EXPECTED DOWNTIME/IMPROVEMENT]
|
||||
|
||||
We can schedule this work at your convenience. Please let us know
|
||||
your preferred maintenance window.
|
||||
|
||||
Best regards,
|
||||
Arizona Computer Guru Support
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Follow-Up Tasks
|
||||
|
||||
- [ ] Contact each critical priority client
|
||||
- [ ] Schedule maintenance windows
|
||||
- [ ] Execute cleanup on arizonahatters.com
|
||||
- [ ] Execute cleanup on peacefulspirit.com
|
||||
- [ ] Implement log rotation across all sites
|
||||
- [ ] Create database maintenance cron job
|
||||
- [ ] Set up monitoring alerts
|
||||
- [ ] Document lessons learned
|
||||
- [ ] Review Wordfence configuration across all sites
|
||||
- [ ] Audit backup retention policies
|
||||
|
||||
---
|
||||
|
||||
## Server Access
|
||||
|
||||
```bash
|
||||
# External SSH
|
||||
ssh root@ix.azcomputerguru.com
|
||||
|
||||
# Internal SSH
|
||||
ssh root@172.16.3.10
|
||||
|
||||
# WHM
|
||||
https://ix.azcomputerguru.com:2087
|
||||
|
||||
# cPanel (example)
|
||||
https://ix.azcomputerguru.com:2083
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
**Original Report:** `~/claude-projects/IX_SERVER_CRITICAL_ISSUES_2026-01-13.md`
|
||||
|
||||
**Session Logs:**
|
||||
- Various client work sessions documented in `~/claude-projects/session-logs/`
|
||||
|
||||
**Scripts Location:**
|
||||
- `/root/scan_sites.sh`
|
||||
- `/root/check_dbs.sh`
|
||||
- `/root/URGENT_SITE_ISSUES.txt`
|
||||
|
||||
---
|
||||
|
||||
## Project History
|
||||
|
||||
**2026-01-13:** Initial comprehensive server scan and issue documentation
|
||||
**2026-01-22:** Imported to ClaudeTools project tracking system
|
||||
|
||||
---
|
||||
|
||||
**Status:** Documented - Awaiting Action
|
||||
**Owner:** Arizona Computer Guru Operations Team
|
||||
**Next Review:** After critical issues resolved
|
||||
Reference in New Issue
Block a user