From d63c9f3a38595bb52733b54bc48195a04580139d Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Mon, 20 Jul 2026 20:10:28 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20IX=20MySQL-move=20=E2=80=94=20add=20cag?= =?UTF-8?q?efsctl=20--remount-all=20step=20+=20CageFS=20root=20cause?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-attempt of the SSD datadir move succeeded (datadir on /ssd, 8/8 DB sites clean). Real cause of the site errors was CageFS not re-mapping the moved socket into user jails -> caged web users got 2002 'No such file or directory' while root was fine. Fix: cagefsctl --remount-all. Runbook + post-mortem updated; verification now uses --no-defaults + response-body checks. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...026-07-20-ix-mysql-ssd-move-incident-postmortem.md | 7 +++++++ .../2026-07-20-ix-mysql-datadir-move-to-ssd.md | 11 +++++++++-- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/clients/internal-infrastructure/reports/2026-07-20-ix-mysql-ssd-move-incident-postmortem.md b/clients/internal-infrastructure/reports/2026-07-20-ix-mysql-ssd-move-incident-postmortem.md index cabaea43..211f98ed 100644 --- a/clients/internal-infrastructure/reports/2026-07-20-ix-mysql-ssd-move-incident-postmortem.md +++ b/clients/internal-infrastructure/reports/2026-07-20-ix-mysql-ssd-move-incident-postmortem.md @@ -77,5 +77,12 @@ curl -sL "https://data.grabbanddurando.com/" | grep -iE "database|mysqli" && ech ``` Cleanup if NOT re-attempting: once `mount | grep /var/lib/mysql` shows it's on cs-root, `rm -rf /ssd/mysql`. +## Update (same day) — re-attempt succeeded, and the REAL cause of the *site* errors: CageFS +The SSD move was re-done and **completed successfully** (datadir on `/ssd`, 143 DBs, per-user auth OK, **8/8 DB sites verified clean by body**). Critically, the re-attempt exposed the genuine technical cause of the site errors that I'd misattributed: + +**This is a CloudLinux CageFS server.** Each user runs in a jailed filesystem, and CageFS maps `/var/lib/mysql` (with `mysql.sock`) into every jail. When the datadir moved to the `/ssd` bind mount, **the jails kept the stale mapping**, so caged web users got `mysqli_connect(): (HY000/2002): No such file or directory` — while **root and the raw socket were fine** (which is exactly why my root-shell `php -r` connected but the sites didn't). The fix: **`cagefsctl --remount-all`** after the move. Sites went clean immediately. + +So the incident had **two independent problems**: (1) the real one — CageFS socket re-mapping (missing `cagefsctl --remount-all`), which caused the actual site outages in *both* attempts; and (2) the self-inflicted `.my.cnf` test artifact, which only entered the picture because the site errors led me to CLI auth-testing. **Had `cagefsctl --remount-all` been in the runbook, the first move would have worked and none of the auth rabbit-hole would have happened.** The runbook now includes that step. + ## MultiAI note Root cause was found by direct testing (`--no-defaults`). AGY (Gemini 3.1 Pro) served as the post-mortem reviewer over two rounds and validated the root cause + authored the prevention/go-forward guidance above. Grok (xAI) was unavailable the entire session (empty results). This is logged as friction in `errorlog.md` (`.my.cnf overrides MYSQL_PWD`). diff --git a/clients/internal-infrastructure/runbooks/2026-07-20-ix-mysql-datadir-move-to-ssd.md b/clients/internal-infrastructure/runbooks/2026-07-20-ix-mysql-datadir-move-to-ssd.md index 7723b12e..8ed64f9f 100644 --- a/clients/internal-infrastructure/runbooks/2026-07-20-ix-mysql-datadir-move-to-ssd.md +++ b/clients/internal-infrastructure/runbooks/2026-07-20-ix-mysql-datadir-move-to-ssd.md @@ -61,6 +61,12 @@ echo '/ssd/mysql /var/lib/mysql none bind 0 0' >> /etc/fstab # persist across # 4) Start MariaDB, then Governor /usr/local/cpanel/scripts/restartsrv_mysql --start systemctl start db_governor + +# 5) CRITICAL on a CageFS server: re-map the moved socket into every user jail. +# Without this, caged web users get "mysqli_connect(): (HY000/2002): No such +# file or directory" even though root + the socket are fine. (Learned the hard +# way 2026-07-20 — see the incident post-mortem.) +cagefsctl --remount-all ``` ## Verification (all must pass) @@ -71,8 +77,9 @@ df -h /var/lib/mysql | tail -1 # /dev/sdc1 (now on SSD mysql -e "SELECT VERSION(); SELECT COUNT(*) AS dbs FROM information_schema.schemata;" mysql -e "SHOW DATABASES;" | head # DBs present ls -l /var/lib/mysql/mysql.sock # socket back in place -/usr/local/cpanel/scripts/restartsrv_mysql --status # cPanel sees it healthy -# Spot-check 2-3 DB-driven sites return HTTP 200 (e.g. curl -sI https:///) +mysql --no-defaults -u -p'' -e "SELECT 1;" # per-user auth (NOT via MYSQL_PWD — /root/.my.cnf overrides it) +# Verify sites by BODY, not status (a PHP DB error returns HTTP 200 with the error in the body): +curl -sL "https:///" | grep -iE "mysqli_connect|database|HY000/200|Access denied" && echo FAIL || echo PASS ``` If all green → success. Leave `/var/lib/mysql.old` in place for a soak period.