diff --git a/clients/internal-infrastructure/reports/2026-07-20-ix-mysql-ssd-move-incident-postmortem.md b/clients/internal-infrastructure/reports/2026-07-20-ix-mysql-ssd-move-incident-postmortem.md new file mode 100644 index 00000000..cabaea43 --- /dev/null +++ b/clients/internal-infrastructure/reports/2026-07-20-ix-mysql-ssd-move-incident-postmortem.md @@ -0,0 +1,81 @@ +# Post-Mortem — IX MariaDB SSD datadir move & the phantom "auth broken" incident + +**Date:** 2026-07-20 · **Server:** ix.azcomputerguru.com (CloudLinux 9, MariaDB 10.11.18 cll-lve, cPanel/WHM, PERC H730P) +**Operator:** Mike / GURU-5070 (Claude main) · **Review:** MultiAI — AGY / Gemini 3.1 Pro (Grok unavailable all session) +**Severity:** Medium (self-inflicted; brief real outages, hours of misdiagnosis, an intended optimization reverted). No data loss. + +## Summary +We successfully moved the MariaDB datadir onto a new SSD RAID1 (bind-mount method), then a client site (ezfast) reported a DB error. Diagnosing it, the operator concluded DB **password/grant changes weren't taking effect** and spent hours chasing phantom MariaDB faults (datadir move, Aria corruption, a missing `auth_socket.so`, the MySQL Governor, broken `FLUSH PRIVILEGES`), including several production restarts and a full rollback of the SSD move. **The actual root cause was a flawed test method, not any DB fault.** The database was healthy the entire time. + +## Root cause +`/root/.my.cnf` contains `[client]` with **`password=`** and **`user=root`**. MySQL client credential precedence is **command-line `-p` > option files (`.my.cnf`) > `MYSQL_PWD` env**. So: +- Every `MYSQL_PWD='knownpw' mysql -u testuser -h localhost` actually authenticated **as `testuser` using root's password** → guaranteed `Access denied (using password: YES)`. +- (Conversely, `mysql --no-defaults` for a *root* query strips root's saved creds → `Access denied (using password: NO)`.) + +This produced a false, perfectly consistent "new/changed users can't authenticate" signal that was misread as grant-subsystem corruption. **Proof:** `mysql --no-defaults -u -p` → **OK**; the `MYSQL_PWD` method → Access denied. Grants, plugins, and the real app users were healthy throughout. + +## Contributing factors (why a test artifact escalated) +1. **Wrong auth-test command** — used `MYSQL_PWD`/bare `mysql -u` from the root shell, where `.my.cnf` silently overrides. Never used `--no-defaults`. +2. **Confused HTTP 200 with "healthy"** — PHP `die(mysqli_connect_error())` returns **200 with the error in the body**. Checked status codes, not bodies, so error pages looked like working sites. +3. **Didn't sanity-check a known-good existing user early** — one correct `--no-defaults` test of any working user would have exposed the artifact in minutes. +4. **Escalated on production mid-diagnosis** — multiple MariaDB restarts + a datadir rollback while the diagnosis was still unproven, causing transient socket-`2002` outages. +5. **Confirmation-bias theory-hopping** — moved between four unrelated root-cause theories, each partially "supported" by the same flawed test. + +## What was NOT the cause (ruled out) +- The SSD datadir move / bind mount (existing users worked; move was clean). +- Aria/`mysql.global_priv` corruption (`CHECK TABLE` = OK). +- The missing `auth_socket.so` plugin (removing it changed nothing — a real-but-harmless misconfig). +- The CloudLinux MySQL Governor (failed identically with it stopped). +- `FLUSH PRIVILEGES` / ACL reload (worked fine). + +## Impact +- Several unnecessary production MariaDB restarts; brief socket-`2002` outages during restart/rollback windows. +- The SSD datadir optimization was rolled back (datadir now on cs-root again; `/ssd` VD + mount remain, unused). +- ezfast had one *genuine, minor* issue — a DB-user password mismatch — fixed correctly once the test flaw was understood. +- The `umount` during rollback was blocked by **`ossec-hids`** holding the MySQL `.err` log (a trap for next time). + +## Resolution / current state +- Datadir rolled back to cs-root; MariaDB healthy; grabb, ezfast, and app users verified via `--no-defaults` and **response-body** checks. +- `my.cnf`: the dead `plugin-load-add = auth_socket.so` line left commented (correct — the `.so` doesn't exist, `unix_socket` is disabled, native-password auth is active). +- Leftover `/ssd/mysql` copy remains on the SSD (harmless; to be cleared before any re-attempt). + +## Prevention checklist (adopt) +```bash +# Root admin CLI (uses /root/.my.cnf on purpose): +mysql -e "SELECT @@datadir;" + +# Test a SPECIFIC user's auth — ALWAYS bypass .my.cnf: +mysql --no-defaults -u -p'' -e "SELECT 1;" + +# Verify a web app's DB health by BODY, never status alone: +curl -sL "https:///" | grep -iE "database|SQLSTATE|Access denied|mysqli_connect|No such file" && echo "FAIL" || echo "PASS" +``` +**Golden rule:** never test per-user/app DB credentials from the root shell without `--no-defaults` — ambient `/root/.my.cnf` defaults silently override `MYSQL_PWD` and even `-u`. And a false, *perfectly consistent* "Access denied" for every test user is a tell for a client-side config trap, not server corruption. **Sanity-check one known-good existing user before theorizing about the server.** + +## Go-forward — re-attempting the SSD move (when scheduled) +Bind-mount rsync is still the right method (fast, ~10 s rollback). Corrected procedure adds the monitoring-daemon pause (the `umount`-busy fix) and correct verification: +```bash +# clear the stale copy first +rm -rf /ssd/mysql/* +# stop DB + the daemons that hold the datadir open +systemctl stop db_governor governor_sentry_daemon +/usr/local/cpanel/scripts/restartsrv_mysql --stop +systemctl stop ossec-hids # <-- the missing step that blocked umount +# sync + bind +rsync -aHAX /var/lib/mysql/ /ssd/mysql/ && chown -R mysql:mysql /ssd/mysql +mv /var/lib/mysql /var/lib/mysql.old && mkdir /var/lib/mysql && chown mysql:mysql /var/lib/mysql && chmod 751 /var/lib/mysql +mount --bind /ssd/mysql /var/lib/mysql +echo '/ssd/mysql /var/lib/mysql none bind 0 0' >> /etc/fstab +# start +/usr/local/cpanel/scripts/restartsrv_mysql --start +systemctl start db_governor governor_sentry_daemon ossec-hids +# VERIFY (the 2-minute suite that would have caught this incident): +findmnt /var/lib/mysql | grep /ssd/mysql +mysql -e "SELECT @@datadir;" # root admin +mysql --no-defaults -u grabblaw_gddata -p'' grabblaw_gdapp_data -e "SELECT 1;" # per-user +curl -sL "https://data.grabbanddurando.com/" | grep -iE "database|mysqli" && echo FAIL || echo PASS # body +``` +Cleanup if NOT re-attempting: once `mount | grep /var/lib/mysql` shows it's on cs-root, `rm -rf /ssd/mysql`. + +## MultiAI note +Root cause was found by direct testing (`--no-defaults`). AGY (Gemini 3.1 Pro) served as the post-mortem reviewer over two rounds and validated the root cause + authored the prevention/go-forward guidance above. Grok (xAI) was unavailable the entire session (empty results). This is logged as friction in `errorlog.md` (`.my.cnf overrides MYSQL_PWD`).