diff --git a/clients/internal-infrastructure/runbooks/2026-07-20-ix-mysql-datadir-move-to-ssd.md b/clients/internal-infrastructure/runbooks/2026-07-20-ix-mysql-datadir-move-to-ssd.md new file mode 100644 index 00000000..7723b12e --- /dev/null +++ b/clients/internal-infrastructure/runbooks/2026-07-20-ix-mysql-datadir-move-to-ssd.md @@ -0,0 +1,104 @@ +# Runbook — Move IX MariaDB datadir to the SSD tier (bind-mount) + +**Server:** ix.azcomputerguru.com (internal 172.16.3.10) · CloudLinux 9 · cPanel +**Goal:** relocate `/var/lib/mysql` (5.3 GB) from the 10K SAS mirror (`cs-root`, `/`) onto the new SSD RAID1 (`/ssd`, `/dev/sdc1`) — for the ~45× random-read latency win. +**Method:** **bind mount** — the datadir *path* stays `/var/lib/mysql`, so cPanel, MySQL Governor, the socket (`/var/lib/mysql/mysql.sock`), and all apps are unaffected. No `my.cnf` change. +**Prepared:** 2026-07-20 (Mike/GURU-5070). Run as `root` on IX. + +## Environment (verified 2026-07-20) +- MariaDB **10.11.18**, service `mariadb.service` +- **CloudLinux MySQL Governor active** — `db_governor.service` + `governor_sentry_daemon.service` (governor-mysql 1.2-147) +- **SELinux: Disabled** — no context relabel needed +- `my.cnf`: no explicit `datadir`/`socket` (compiled defaults → datadir `/var/lib/mysql`, socket `/var/lib/mysql/mysql.sock`) +- datadir owner `mysql:mysql`, mode `751` +- Target `/ssd` (XFS, noatime) = `/dev/sdc1`, 1.5 TB, ~1% used +- cPanel wrapper: `/usr/local/cpanel/scripts/restartsrv_mysql` + +## Downtime +~**2–5 minutes** of DB downtime (stop → rsync 5.3 GB → start). DB-driven sites are down during the window. **Run off-hours.** + +## Safety posture +- The rsync leaves the **original datadir intact** as `/var/lib/mysql.old` (full cold copy = instant rollback + backup). Data ends up in **two** places until cleanup. +- Plus an optional pre-move logical dump. +- SELinux disabled + path-unchanged bind mount = no cPanel/Governor/socket/app changes. + +--- + +## Pre-flight +```bash +# Confirm the SSD mount + free space, current datadir device, services up +findmnt /ssd && df -h /ssd | tail -1 +df -h /var/lib/mysql | tail -1 # currently on /dev/mapper/cs-root +systemctl is-active mariadb db_governor # both should be 'active' +du -sh /var/lib/mysql # ~5.3G + +# OPTIONAL belt-and-suspenders logical dump (while DB is still up) +mysqldump --all-databases --routines --events --triggers \ + --single-transaction --quick > /root/premove-alldb-$(date +%F).sql +ls -lh /root/premove-alldb-*.sql +``` + +## Execution +```bash +# 1) Stop Governor, then MariaDB (cPanel-aware). Confirm down. +systemctl stop db_governor +/usr/local/cpanel/scripts/restartsrv_mysql --stop +systemctl is-active mariadb # expect: inactive/failed(=stopped) + +# 2) Copy datadir to SSD, preserving perms/hardlinks/ACLs/xattrs +rsync -aHAX --info=progress2 /var/lib/mysql/ /ssd/mysql/ +chown mysql:mysql /ssd/mysql +du -sh /var/lib/mysql /ssd/mysql # sizes should match (~5.3G each) + +# 3) Swap: keep original as backup, bind-mount the SSD copy over the same path +mv /var/lib/mysql /var/lib/mysql.old +mkdir /var/lib/mysql +chown mysql:mysql /var/lib/mysql +chmod 751 /var/lib/mysql +mount --bind /ssd/mysql /var/lib/mysql +echo '/ssd/mysql /var/lib/mysql none bind 0 0' >> /etc/fstab # persist across reboot + +# 4) Start MariaDB, then Governor +/usr/local/cpanel/scripts/restartsrv_mysql --start +systemctl start db_governor +``` + +## Verification (all must pass) +```bash +systemctl is-active mariadb db_governor # both active +findmnt /var/lib/mysql # SOURCE = /dev/sdc1[/... ] (bind) +df -h /var/lib/mysql | tail -1 # /dev/sdc1 (now on SSD) +mysql -e "SELECT VERSION(); SELECT COUNT(*) AS dbs FROM information_schema.schemata;" +mysql -e "SHOW DATABASES;" | head # DBs present +ls -l /var/lib/mysql/mysql.sock # socket back in place +/usr/local/cpanel/scripts/restartsrv_mysql --status # cPanel sees it healthy +# Spot-check 2-3 DB-driven sites return HTTP 200 (e.g. curl -sI https:///) +``` +If all green → success. Leave `/var/lib/mysql.old` in place for a soak period. + +## Rollback (if anything is wrong — reverts to the original on cs-root) +```bash +systemctl stop db_governor +/usr/local/cpanel/scripts/restartsrv_mysql --stop +umount /var/lib/mysql +rmdir /var/lib/mysql +mv /var/lib/mysql.old /var/lib/mysql +sed -i '\#^/ssd/mysql /var/lib/mysql #d' /etc/fstab # drop the bind line +/usr/local/cpanel/scripts/restartsrv_mysql --start +systemctl start db_governor +# verify: df -h /var/lib/mysql -> back on /dev/mapper/cs-root +``` +Instant, because the original datadir was never modified. + +## Cleanup (after ~1 week of clean running) +```bash +rm -rf /var/lib/mysql.old # reclaim 5.3 GB on cs-root +rm -f /root/premove-alldb-*.sql +``` + +## Notes / gotchas +- **Bind mount, not a datadir change** — chosen precisely because cPanel + MySQL Governor + hardcoded `/var/lib/mysql/mysql.sock` all keep working with zero config edits. Do **not** also edit `my.cnf datadir`. +- **Governor:** it's a watcher on MariaDB; stopping/starting it around the swap avoids watcher errors mid-swap. It auto-reconnects once MariaDB is back. +- **SELinux disabled** on this host, so no `restorecon`/context work. (If it were enforcing, the bind mount inherits `/var/lib/mysql`'s context anyway.) +- **Disk cache is OFF** on the SSD VD (non-PLP BX500 safety). DB writes commit synchronously (~2.2k IOPS ceiling) — fine for this read-heavy 5.3 GB DB. Do not enable disk cache to chase write speed without accepting the power-loss trade-off. +- Reboot-safe: the fstab bind line remounts `/ssd/mysql` → `/var/lib/mysql` on boot (after `/ssd` mounts, which is also in fstab).