runbook: IX MariaDB datadir move to SSD tier (bind-mount)

Bind-mount method (datadir path stays /var/lib/mysql -> physically on /ssd),
transparent to cPanel + MySQL Governor + socket. Verified IX specifics
(MariaDB 10.11.18, SELinux disabled, governor active). Pre-flight, execution,
verification gates, instant rollback, cleanup.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

clients/internal-infrastructure/runbooks/2026-07-20-ix-mysql-datadir-move-to-ssd.md

@@ -0,0 +1,104 @@
+# Runbook — Move IX MariaDB datadir to the SSD tier (bind-mount)
+
+**Server:** ix.azcomputerguru.com (internal 172.16.3.10) · CloudLinux 9 · cPanel
+**Goal:** relocate `/var/lib/mysql` (5.3 GB) from the 10K SAS mirror (`cs-root`, `/`) onto the new SSD RAID1 (`/ssd`, `/dev/sdc1`) — for the ~45× random-read latency win.
+**Method:** **bind mount** — the datadir *path* stays `/var/lib/mysql`, so cPanel, MySQL Governor, the socket (`/var/lib/mysql/mysql.sock`), and all apps are unaffected. No `my.cnf` change.
+**Prepared:** 2026-07-20 (Mike/GURU-5070). Run as `root` on IX.
+
+## Environment (verified 2026-07-20)
+- MariaDB **10.11.18**, service `mariadb.service`
+- **CloudLinux MySQL Governor active** — `db_governor.service` + `governor_sentry_daemon.service` (governor-mysql 1.2-147)
+- **SELinux: Disabled** — no context relabel needed
+- `my.cnf`: no explicit `datadir`/`socket` (compiled defaults → datadir `/var/lib/mysql`, socket `/var/lib/mysql/mysql.sock`)
+- datadir owner `mysql:mysql`, mode `751`
+- Target `/ssd` (XFS, noatime) = `/dev/sdc1`, 1.5 TB, ~1% used
+- cPanel wrapper: `/usr/local/cpanel/scripts/restartsrv_mysql`
+
+## Downtime
+~**2–5 minutes** of DB downtime (stop → rsync 5.3 GB → start). DB-driven sites are down during the window. **Run off-hours.**
+
+## Safety posture
+- The rsync leaves the **original datadir intact** as `/var/lib/mysql.old` (full cold copy = instant rollback + backup). Data ends up in **two** places until cleanup.
+- Plus an optional pre-move logical dump.
+- SELinux disabled + path-unchanged bind mount = no cPanel/Governor/socket/app changes.
+
+---
+
+## Pre-flight
+```bash
+# Confirm the SSD mount + free space, current datadir device, services up
+findmnt /ssd && df -h /ssd | tail -1
+df -h /var/lib/mysql | tail -1                 # currently on /dev/mapper/cs-root
+systemctl is-active mariadb db_governor        # both should be 'active'
+du -sh /var/lib/mysql                          # ~5.3G
+
+# OPTIONAL belt-and-suspenders logical dump (while DB is still up)
+mysqldump --all-databases --routines --events --triggers \
+  --single-transaction --quick > /root/premove-alldb-$(date +%F).sql
+ls -lh /root/premove-alldb-*.sql
+```
+
+## Execution
+```bash
+# 1) Stop Governor, then MariaDB (cPanel-aware). Confirm down.
+systemctl stop db_governor
+/usr/local/cpanel/scripts/restartsrv_mysql --stop
+systemctl is-active mariadb        # expect: inactive/failed(=stopped)
+
+# 2) Copy datadir to SSD, preserving perms/hardlinks/ACLs/xattrs
+rsync -aHAX --info=progress2 /var/lib/mysql/ /ssd/mysql/
+chown mysql:mysql /ssd/mysql
+du -sh /var/lib/mysql /ssd/mysql    # sizes should match (~5.3G each)
+
+# 3) Swap: keep original as backup, bind-mount the SSD copy over the same path
+mv /var/lib/mysql /var/lib/mysql.old
+mkdir /var/lib/mysql
+chown mysql:mysql /var/lib/mysql
+chmod 751 /var/lib/mysql
+mount --bind /ssd/mysql /var/lib/mysql
+echo '/ssd/mysql /var/lib/mysql none bind 0 0' >> /etc/fstab   # persist across reboot
+
+# 4) Start MariaDB, then Governor
+/usr/local/cpanel/scripts/restartsrv_mysql --start
+systemctl start db_governor
+```
+
+## Verification (all must pass)
+```bash
+systemctl is-active mariadb db_governor                 # both active
+findmnt /var/lib/mysql                                  # SOURCE = /dev/sdc1[/... ] (bind)
+df -h /var/lib/mysql | tail -1                          # /dev/sdc1  (now on SSD)
+mysql -e "SELECT VERSION(); SELECT COUNT(*) AS dbs FROM information_schema.schemata;"
+mysql -e "SHOW DATABASES;" | head                       # DBs present
+ls -l /var/lib/mysql/mysql.sock                         # socket back in place
+/usr/local/cpanel/scripts/restartsrv_mysql --status     # cPanel sees it healthy
+# Spot-check 2-3 DB-driven sites return HTTP 200 (e.g. curl -sI https://<site>/)
+```
+If all green → success. Leave `/var/lib/mysql.old` in place for a soak period.
+
+## Rollback (if anything is wrong — reverts to the original on cs-root)
+```bash
+systemctl stop db_governor
+/usr/local/cpanel/scripts/restartsrv_mysql --stop
+umount /var/lib/mysql
+rmdir /var/lib/mysql
+mv /var/lib/mysql.old /var/lib/mysql
+sed -i '\#^/ssd/mysql /var/lib/mysql #d' /etc/fstab      # drop the bind line
+/usr/local/cpanel/scripts/restartsrv_mysql --start
+systemctl start db_governor
+# verify: df -h /var/lib/mysql -> back on /dev/mapper/cs-root
+```
+Instant, because the original datadir was never modified.
+
+## Cleanup (after ~1 week of clean running)
+```bash
+rm -rf /var/lib/mysql.old              # reclaim 5.3 GB on cs-root
+rm -f /root/premove-alldb-*.sql
+```
+
+## Notes / gotchas
+- **Bind mount, not a datadir change** — chosen precisely because cPanel + MySQL Governor + hardcoded `/var/lib/mysql/mysql.sock` all keep working with zero config edits. Do **not** also edit `my.cnf datadir`.
+- **Governor:** it's a watcher on MariaDB; stopping/starting it around the swap avoids watcher errors mid-swap. It auto-reconnects once MariaDB is back.
+- **SELinux disabled** on this host, so no `restorecon`/context work. (If it were enforcing, the bind mount inherits `/var/lib/mysql`'s context anyway.)
+- **Disk cache is OFF** on the SSD VD (non-PLP BX500 safety). DB writes commit synchronously (~2.2k IOPS ceiling) — fine for this read-heavy 5.3 GB DB. Do not enable disk cache to chase write speed without accepting the power-loss trade-off.
+- Reboot-safe: the fstab bind line remounts `/ssd/mysql` → `/var/lib/mysql` on boot (after `/ssd` mounts, which is also in fstab).