postmortem: IX MariaDB SSD-move phantom "auth broken" incident
Root cause was a testing artifact, not a DB fault: /root/.my.cnf (password + user=root) overrides MYSQL_PWD, so every 'MYSQL_PWD=x mysql -u testuser' sent root's password -> false Access denied. Chased 4 wrong theories + unnecessary prod restarts + a datadir rollback before finding it. DB was healthy throughout. Includes prevention checklist (--no-defaults for per-user tests, body checks not status codes) + corrected SSD re-attempt procedure (ossec-hids pause). Reviewed via MultiAI (AGY; Grok unavailable). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,81 @@
|
|||||||
|
# Post-Mortem — IX MariaDB SSD datadir move & the phantom "auth broken" incident
|
||||||
|
|
||||||
|
**Date:** 2026-07-20 · **Server:** ix.azcomputerguru.com (CloudLinux 9, MariaDB 10.11.18 cll-lve, cPanel/WHM, PERC H730P)
|
||||||
|
**Operator:** Mike / GURU-5070 (Claude main) · **Review:** MultiAI — AGY / Gemini 3.1 Pro (Grok unavailable all session)
|
||||||
|
**Severity:** Medium (self-inflicted; brief real outages, hours of misdiagnosis, an intended optimization reverted). No data loss.
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
We successfully moved the MariaDB datadir onto a new SSD RAID1 (bind-mount method), then a client site (ezfast) reported a DB error. Diagnosing it, the operator concluded DB **password/grant changes weren't taking effect** and spent hours chasing phantom MariaDB faults (datadir move, Aria corruption, a missing `auth_socket.so`, the MySQL Governor, broken `FLUSH PRIVILEGES`), including several production restarts and a full rollback of the SSD move. **The actual root cause was a flawed test method, not any DB fault.** The database was healthy the entire time.
|
||||||
|
|
||||||
|
## Root cause
|
||||||
|
`/root/.my.cnf` contains `[client]` with **`password=`** and **`user=root`**. MySQL client credential precedence is **command-line `-p` > option files (`.my.cnf`) > `MYSQL_PWD` env**. So:
|
||||||
|
- Every `MYSQL_PWD='knownpw' mysql -u testuser -h localhost` actually authenticated **as `testuser` using root's password** → guaranteed `Access denied (using password: YES)`.
|
||||||
|
- (Conversely, `mysql --no-defaults` for a *root* query strips root's saved creds → `Access denied (using password: NO)`.)
|
||||||
|
|
||||||
|
This produced a false, perfectly consistent "new/changed users can't authenticate" signal that was misread as grant-subsystem corruption. **Proof:** `mysql --no-defaults -u <user> -p<pass>` → **OK**; the `MYSQL_PWD` method → Access denied. Grants, plugins, and the real app users were healthy throughout.
|
||||||
|
|
||||||
|
## Contributing factors (why a test artifact escalated)
|
||||||
|
1. **Wrong auth-test command** — used `MYSQL_PWD`/bare `mysql -u` from the root shell, where `.my.cnf` silently overrides. Never used `--no-defaults`.
|
||||||
|
2. **Confused HTTP 200 with "healthy"** — PHP `die(mysqli_connect_error())` returns **200 with the error in the body**. Checked status codes, not bodies, so error pages looked like working sites.
|
||||||
|
3. **Didn't sanity-check a known-good existing user early** — one correct `--no-defaults` test of any working user would have exposed the artifact in minutes.
|
||||||
|
4. **Escalated on production mid-diagnosis** — multiple MariaDB restarts + a datadir rollback while the diagnosis was still unproven, causing transient socket-`2002` outages.
|
||||||
|
5. **Confirmation-bias theory-hopping** — moved between four unrelated root-cause theories, each partially "supported" by the same flawed test.
|
||||||
|
|
||||||
|
## What was NOT the cause (ruled out)
|
||||||
|
- The SSD datadir move / bind mount (existing users worked; move was clean).
|
||||||
|
- Aria/`mysql.global_priv` corruption (`CHECK TABLE` = OK).
|
||||||
|
- The missing `auth_socket.so` plugin (removing it changed nothing — a real-but-harmless misconfig).
|
||||||
|
- The CloudLinux MySQL Governor (failed identically with it stopped).
|
||||||
|
- `FLUSH PRIVILEGES` / ACL reload (worked fine).
|
||||||
|
|
||||||
|
## Impact
|
||||||
|
- Several unnecessary production MariaDB restarts; brief socket-`2002` outages during restart/rollback windows.
|
||||||
|
- The SSD datadir optimization was rolled back (datadir now on cs-root again; `/ssd` VD + mount remain, unused).
|
||||||
|
- ezfast had one *genuine, minor* issue — a DB-user password mismatch — fixed correctly once the test flaw was understood.
|
||||||
|
- The `umount` during rollback was blocked by **`ossec-hids`** holding the MySQL `.err` log (a trap for next time).
|
||||||
|
|
||||||
|
## Resolution / current state
|
||||||
|
- Datadir rolled back to cs-root; MariaDB healthy; grabb, ezfast, and app users verified via `--no-defaults` and **response-body** checks.
|
||||||
|
- `my.cnf`: the dead `plugin-load-add = auth_socket.so` line left commented (correct — the `.so` doesn't exist, `unix_socket` is disabled, native-password auth is active).
|
||||||
|
- Leftover `/ssd/mysql` copy remains on the SSD (harmless; to be cleared before any re-attempt).
|
||||||
|
|
||||||
|
## Prevention checklist (adopt)
|
||||||
|
```bash
|
||||||
|
# Root admin CLI (uses /root/.my.cnf on purpose):
|
||||||
|
mysql -e "SELECT @@datadir;"
|
||||||
|
|
||||||
|
# Test a SPECIFIC user's auth — ALWAYS bypass .my.cnf:
|
||||||
|
mysql --no-defaults -u <user> -p'<pass>' <db> -e "SELECT 1;"
|
||||||
|
|
||||||
|
# Verify a web app's DB health by BODY, never status alone:
|
||||||
|
curl -sL "https://<site>/<subpage>" | grep -iE "database|SQLSTATE|Access denied|mysqli_connect|No such file" && echo "FAIL" || echo "PASS"
|
||||||
|
```
|
||||||
|
**Golden rule:** never test per-user/app DB credentials from the root shell without `--no-defaults` — ambient `/root/.my.cnf` defaults silently override `MYSQL_PWD` and even `-u`. And a false, *perfectly consistent* "Access denied" for every test user is a tell for a client-side config trap, not server corruption. **Sanity-check one known-good existing user before theorizing about the server.**
|
||||||
|
|
||||||
|
## Go-forward — re-attempting the SSD move (when scheduled)
|
||||||
|
Bind-mount rsync is still the right method (fast, ~10 s rollback). Corrected procedure adds the monitoring-daemon pause (the `umount`-busy fix) and correct verification:
|
||||||
|
```bash
|
||||||
|
# clear the stale copy first
|
||||||
|
rm -rf /ssd/mysql/*
|
||||||
|
# stop DB + the daemons that hold the datadir open
|
||||||
|
systemctl stop db_governor governor_sentry_daemon
|
||||||
|
/usr/local/cpanel/scripts/restartsrv_mysql --stop
|
||||||
|
systemctl stop ossec-hids # <-- the missing step that blocked umount
|
||||||
|
# sync + bind
|
||||||
|
rsync -aHAX /var/lib/mysql/ /ssd/mysql/ && chown -R mysql:mysql /ssd/mysql
|
||||||
|
mv /var/lib/mysql /var/lib/mysql.old && mkdir /var/lib/mysql && chown mysql:mysql /var/lib/mysql && chmod 751 /var/lib/mysql
|
||||||
|
mount --bind /ssd/mysql /var/lib/mysql
|
||||||
|
echo '/ssd/mysql /var/lib/mysql none bind 0 0' >> /etc/fstab
|
||||||
|
# start
|
||||||
|
/usr/local/cpanel/scripts/restartsrv_mysql --start
|
||||||
|
systemctl start db_governor governor_sentry_daemon ossec-hids
|
||||||
|
# VERIFY (the 2-minute suite that would have caught this incident):
|
||||||
|
findmnt /var/lib/mysql | grep /ssd/mysql
|
||||||
|
mysql -e "SELECT @@datadir;" # root admin
|
||||||
|
mysql --no-defaults -u grabblaw_gddata -p'<pass>' grabblaw_gdapp_data -e "SELECT 1;" # per-user
|
||||||
|
curl -sL "https://data.grabbanddurando.com/" | grep -iE "database|mysqli" && echo FAIL || echo PASS # body
|
||||||
|
```
|
||||||
|
Cleanup if NOT re-attempting: once `mount | grep /var/lib/mysql` shows it's on cs-root, `rm -rf /ssd/mysql`.
|
||||||
|
|
||||||
|
## MultiAI note
|
||||||
|
Root cause was found by direct testing (`--no-defaults`). AGY (Gemini 3.1 Pro) served as the post-mortem reviewer over two rounds and validated the root cause + authored the prevention/go-forward guidance above. Grok (xAI) was unavailable the entire session (empty results). This is logged as friction in `errorlog.md` (`.my.cnf overrides MYSQL_PWD`).
|
||||||
Reference in New Issue
Block a user