azcomputerguru/claudetools
1
0
Fork 0
Code Issues 59 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: IX MySQL-move — add cagefsctl --remount-all step + CageFS root cause

Browse Source 
Re-attempt of the SSD datadir move succeeded (datadir on /ssd, 8/8 DB sites
clean). Real cause of the site errors was CageFS not re-mapping the moved socket
into user jails -> caged web users got 2002 'No such file or directory' while
root was fine. Fix: cagefsctl --remount-all. Runbook + post-mortem updated;
verification now uses --no-defaults + response-body checks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Mike Swanson
2026-07-20 20:10:28 -07:00
parent 62b113270c
commit d63c9f3a38
2 changed files with 16 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
clients/internal-infrastructure/reports/2026-07-20-ix-mysql-ssd-move-incident-postmortem.md
View File

@@ -77,5 +77,12 @@ curl -sL "https://data.grabbanddurando.com/" | grep -iE "database|mysqli" && ech
```
Cleanup if NOT re-attempting: once `mount | grep /var/lib/mysql` shows it's on cs-root, `rm -rf /ssd/mysql`.
## Update (same day) — re-attempt succeeded, and the REAL cause of the *site* errors: CageFS
The SSD move was re-done and **completed successfully** (datadir on `/ssd`, 143 DBs, per-user auth OK, **8/8 DB sites verified clean by body**). Critically, the re-attempt exposed the genuine technical cause of the site errors that I'd misattributed:
**This is a CloudLinux CageFS server.** Each user runs in a jailed filesystem, and CageFS maps `/var/lib/mysql` (with `mysql.sock`) into every jail. When the datadir moved to the `/ssd` bind mount, **the jails kept the stale mapping**, so caged web users got `mysqli_connect(): (HY000/2002): No such file or directory` — while **root and the raw socket were fine** (which is exactly why my root-shell `php -r` connected but the sites didn't). The fix: **`cagefsctl --remount-all`** after the move. Sites went clean immediately.
So the incident had **two independent problems**: (1) the real one — CageFS socket re-mapping (missing `cagefsctl --remount-all`), which caused the actual site outages in *both* attempts; and (2) the self-inflicted `.my.cnf` test artifact, which only entered the picture because the site errors led me to CLI auth-testing. **Had `cagefsctl --remount-all` been in the runbook, the first move would have worked and none of the auth rabbit-hole would have happened.** The runbook now includes that step.
## MultiAI note
Root cause was found by direct testing (`--no-defaults`). AGY (Gemini 3.1 Pro) served as the post-mortem reviewer over two rounds and validated the root cause + authored the prevention/go-forward guidance above. Grok (xAI) was unavailable the entire session (empty results). This is logged as friction in `errorlog.md` (`.my.cnf overrides MYSQL_PWD`).

11
clients/internal-infrastructure/runbooks/2026-07-20-ix-mysql-datadir-move-to-ssd.md
View File

@@ -61,6 +61,12 @@ echo '/ssd/mysql /var/lib/mysql none bind 0 0' >> /etc/fstab # persist across
# 4) Start MariaDB, then Governor
/usr/local/cpanel/scripts/restartsrv_mysql --start
systemctl start db_governor
# 5) CRITICAL on a CageFS server: re-map the moved socket into every user jail.
# Without this, caged web users get "mysqli_connect(): (HY000/2002): No such
# file or directory" even though root + the socket are fine. (Learned the hard
# way 2026-07-20 — see the incident post-mortem.)
cagefsctl --remount-all
```
## Verification (all must pass)
@@ -71,8 +77,9 @@ df -h /var/lib/mysql | tail -1 # /dev/sdc1 (now on SSD
mysql -e "SELECT VERSION(); SELECT COUNT(*) AS dbs FROM information_schema.schemata;"
mysql -e "SHOW DATABASES;" | head # DBs present
ls -l /var/lib/mysql/mysql.sock # socket back in place
/usr/local/cpanel/scripts/restartsrv_mysql --status # cPanel sees it healthy
# Spot-check 2-3 DB-driven sites return HTTP 200 (e.g. curl -sI https://<site>/)
mysql --no-defaults -u <appuser> -p'<pass>' <db> -e "SELECT 1;" # per-user auth (NOT via MYSQL_PWD — /root/.my.cnf overrides it)
# Verify sites by BODY, not status (a PHP DB error returns HTTP 200 with the error in the body):
curl -sL "https://<db-site>/<subpage>" | grep -iE "mysqli_connect|database|HY000/200|Access denied" && echo FAIL || echo PASS
```
If all green → success. Leave `/var/lib/mysql.old` in place for a soak period.