IX: PHP-FPM/OPcache fleet remediation report + ezfast pilot

ezfastautoglass.com moved to ea-php84 + PHP-FPM + OPcache (3.1s -> 0.44s TTFB, ~7x).
Read-only fleet sweep found 71/72 IX sites run PHP per-request (suPHP/CGI or alt-php
Selector) with OPcache cold - the server default. Report documents root cause, evidence,
per-vhost FPM fix, and a phased/verified rollout plan. Also commits the prior web-tier
SSD plan and a project memory of the finding.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.claude/memory/MEMORY.md

@@ -25,6 +25,7 @@
 - [Firefox driver (ff.py)](reference_ff_firefox_driver.md) — PREFERRED browser driver. Drive Firefox via Playwright (.claude/scripts/ff.py): daemon on :9333, persistent profile, nav/shot/click/type/eval/console/network. Mike dislikes Chrome; claude-in-chrome connector disabled 2026-06-06.
 - [Community Forum (Flarum)](reference_community_forum.md) — Flarum forum at community.azcomputerguru.com, API access, database, posting workflow.
 - [Radio Show Website](reference_radio_website.md) — Astro static site at radio.azcomputerguru.com on IX server.
+- [IX PHP-FPM/OPcache remediation](project_ix_php_fpm_opcache_remediation.md) — IX server default is suPHP/CGI per-request with OPcache cold: 71/72 sites recompile PHP every request (1-5.6s TTFB). Fix = per-vhost PHP-FPM. ezfast pilot done (3.1s->0.44s); fleet rollout proposed.
 - [IX Server Access](reference_ix_server_access.md) — `ix.azcomputerguru.com` / 172.16.3.10. Reachable when Tailscale is on (no VPN). SSH currently uses sshpass with root password; key auth from GURU-5070 not configured yet (was CachyOS, now Win11 — verify).
 - [Cloudflare access](reference_cloudflare_access.md) — Cloudflare API creds in SOPS `services/cloudflare.sops.yaml` (full DNS + account tokens; azcomputerguru zone_id 1beb9917...). azcomputerguru.com DNS is on Cloudflare (not IX) — edit via Cloudflare API, not whmapi1.
 - [Matomo Analytics](reference_matomo_analytics.md) — Self-hosted analytics at analytics.azcomputerguru.com, site IDs, tracking for all 3 sites.

.claude/memory/project_ix_php_fpm_opcache_remediation.md

@@ -0,0 +1,36 @@
+---
+name: project_ix_php_fpm_opcache_remediation
+description: IX fleet-wide PHP is per-request with OPcache cold (server default); fix is per-vhost PHP-FPM. ezfast pilot done.
+metadata:
+  type: project
+---
+
+On **ix.azcomputerguru.com** (CloudLinux 9 + CageFS + cPanel), the **server default PHP handler is
+suPHP/CGI** (per-request), and 10 accounts run **CloudLinux PHP Selector alt-phpXX** (also per-request).
+Result discovered 2026-07-20: **71 of 72 unsuspended sites recompile PHP on every request with OPcache
+effectively cold** — OPcache SHM can't persist without a persistent SAPI. The `ea-phpXX-php-opcache`
+packages are already installed+enabled for ea-php81/82/84/74, just wasted. This makes nearly every WP
+site 1-5.6s TTFB purely from bytecode recompilation (not disk — DB is already on SSD, not the DB, not
+worker starvation).
+
+**Fix = per-vhost PHP-FPM.** For suPHP/CGI sites just `whmapi1 php_set_vhost_versions version=<current
+ea-phpXX> vhost=<domain> php_fpm=1` (OPcache becomes effective immediately). For alt-php Selector sites:
+`selectorctl --set-user-current=native --user=<user>` then set ea-php84 + FPM. `.htaccess`
+php_value/php_flag are `<IfModule>`-guarded and do NOT 500 under FPM (mod_lsapi stays loaded, registers
+the directives, blocks go inert; FPM reads the auto-generated `.user.ini`).
+
+**Diagnosis proof:** SAPI probe file inside a live request = definitive (`php_sapi_name()`, `getmypid()`
+changing per request = per-request spawn, `opcache_get_status()` = cold). A trivial PHP file ~0.13s vs WP
+homepage ~3s isolates compile/bootstrap as the cost.
+
+**Pilot DONE:** ezfastautoglass.com 3.1s -> 0.44s (ea-php84 + FPM + OPcache; had to `yum install
+ea-php84-php-opcache` first — it was the first ea-php84 FPM site). Raised ea-php84 OPcache limits
+(memory_consumption=256, max_accelerated_files=16229, interned_strings_buffer=32); default 4000 files was
+99.6% full on a 31-plugin WP. validate_timestamps left ON (2s) so edits go live without manual reset.
+
+**Rollout is PROPOSED, phased, per-site verified with body-check + auto-rollback** (NOT a fleet flip).
+Report: `clients/internal-infrastructure/reports/2026-07-20-ix-php-fpm-opcache-remediation.md`. Also file
+separately: hightechmortgage.com live 500; sparcoracing/rarengineer/jparkinsonaz/jefffurrier unreachable.
+Discipline from the same-day SSD-move incident: verify by response BODY not HTTP status; test DB with
+`mysql --no-defaults -u U -pPASS`; `cagefsctl --remount-all` after socket/path moves. See
+[[feedback_tmp_path_windows]] for the base64-pipe pattern used to run these remote scripts safely.

clients/internal-infrastructure/reports/2026-07-20-ix-php-fpm-opcache-remediation.md

@@ -0,0 +1,165 @@
+# Remediation — IX PHP handler / OPcache (fleet-wide) + ezfastautoglass pilot
+
+**Date:** 2026-07-20 · **Server:** ix.azcomputerguru.com (CloudLinux 9 + CageFS + cPanel/WHM, Dell R740xd)
+**Author:** Mike Swanson (GURU-5070) · **Status:** Pilot COMPLETE (ezfastautoglass), fleet rollout PROPOSED (awaiting go)
+**Related:** `2026-07-20-ix-mysql-ssd-move-incident-postmortem.md`, `2026-07-20-ix-web-tier-to-ssd-plan.md`
+
+---
+
+## 1. Executive summary
+
+Chasing a "slow page loads" report on ezfastautoglass.com, a multi-AI handler trace found the site ran
+PHP **per-request with OPcache disabled** — recompiling WordPress core + 31 plugins from source on *every*
+hit (TTFB 3.1s). We moved it to **ea-php84 + PHP-FPM + OPcache** and TTFB dropped to **~0.44s (~7x)**.
+
+A follow-up **read-only sweep of all 72 unsuspended accounts** showed this is **not site-specific — it is the
+server default**. **71 of 72 sites** run PHP through a **per-request SAPI (suPHP/CGI or alt-php Selector) with
+OPcache effectively cold.** Every PHP request on nearly the whole box pays a full compile. 36 sites exceed 1s
+TTFB purely from this; the worst are 5.6s. The OPcache extension is already installed and enabled for the
+common ea-php versions — it is simply wasted because no persistent PHP process exists to hold the cache.
+
+**The fix is cheap and mostly one-flag:** enable PHP-FPM per vhost (`php_fpm=1`). OPcache then becomes
+effective immediately with no package work for ea-php81/82/84/74. Rollout should be **phased and
+per-site-verified** (body check + auto-rollback), not a single fleet flip.
+
+---
+
+## 2. Root cause
+
+| Layer | State on IX | Effect |
+|---|---|---|
+| Default PHP handler | **suPHP / CGI** (cPanel default), or **CloudLinux PHP Selector alt-phpXX** for 10 accounts | PHP process is spawned **per request** and torn down |
+| OPcache | Extension present + `opcache.enable=1` for ea-php81/82/84/74 | SHM opcode cache is created inside each short-lived worker and **discarded when it exits** — so it never warms |
+| Net | Persistent-process SAPI (FPM / mod_lsapi daemon) is **not** in the request path | **Every request recompiles all PHP from disk** |
+
+Confirmed from inside a live request (SAPI probe): `SAPI=cgi-fcgi`, `OPCACHE_ENABLED=NO`, and a **different PID
+on every hit** (8 sequential requests = 8 PIDs). A trivial PHP file returned in 0.13s while the full WP homepage
+took 3.1s — i.e. the cost is bytecode compilation + bootstrap, not the handler spawn, not disk, not the DB
+(the DB is already on SSD), and not worker starvation (concurrency test showed no queuing).
+
+This is a **configuration default**, not a fault: cPanel ships suPHP/CGI as the baseline handler, and the
+10 Selector accounts were switched to alt-phpXX at some point without enabling the OPcache extension in the
+Selector. Neither path keeps a persistent process, so the already-installed OPcache can never do its job.
+
+---
+
+## 3. Evidence — fleet sweep (read-only, 2026-07-20)
+
+72 unsuspended accounts. Classification = effective PHP + handler (Selector override accounted for) +
+whether OPcache can actually persist + live external TTFB.
+
+**Server-wide OPcache availability (already in place):**
+`ea-php81 YES · ea-php82 YES · ea-php84 YES · ea-php74 YES · ea-php83 MISSING (no sites use 83)`
+
+**Summary:**
+- OPcache NOT effective: **71 / 72** (only ezfastautoglass, post-fix, is effective)
+- suPHP/CGI per-request: **61** · alt-php84 Selector per-request: **10**
+- **Fix candidates (TTFB > 1s, OPcache off, HTTP 2xx): 36**
+
+**Worst offenders (TTFB, all OPcache-cold):**
+
+| Site | User | Eff. PHP | Handler | TTFB |
+|---|---|---|---|---|
+| jrkco.com | jrkco | ea-php81 | suphp/cgi | 5.61s |
+| bestmassageintucson.com | peacefulspirit | ea-php74 | suphp/cgi | 5.60s |
+| tucsonadobe.com | tucsonadobe | ea-php82 | suphp/cgi | 4.40s |
+| sandtekomachinery.com | sandtekomachinery | ea-php81 | suphp/cgi | 3.62s |
+| gurushow.com | gurushow | ea-php81 | suphp/cgi | 3.58s |
+| fsgtucson.com | fsgtucson | ea-php74 | suphp/cgi | 3.26s |
+| arizonahatters.com | arizonahatters | ea-php81 | suphp/cgi | 3.14s |
+| acepickupparts.com | acepickupparts | ea-php81 | suphp/cgi | 3.02s |
+| tarafulcreations.com | tarafulcreations | ea-php81 | suphp/cgi | 2.90s |
+| phxpropane.com | phxpropane | ea-php80 | suphp/cgi | 2.79s |
+| themarcgroup.com | themarcgroup | ea-php74 | suphp/cgi | 2.77s |
+| danaise.com | danaise | ea-php81 | suphp/cgi | 2.69s |
+| voicesofthewest.net | voicesofthewest | ea-php81 | suphp/cgi | 2.51s |
+| drsticken.com | drsticken | ea-php80 | suphp/cgi | 2.40s |
+| anti-vaxxer.org | antivaxxer | ea-php81 | suphp/cgi | 2.08s |
+| farwestwell.com | farwest | ea-php81 | suphp/cgi | 2.03s |
+
+(20 more sites between 1.0s and 2.0s — thrive, photonicapps, packetdial, cryoweave, blackswanarchery,
+rednourlaw, cascades, valleydevelopmentcorp, ultraperformance, and others. Full ranked list in the
+session log / sweep output.)
+
+**Not perf, but surfaced by the sweep — track separately:**
+- **HTTP 500 (live error):** hightechmortgage.com (alt-php84) — a real broken site, not just slow.
+- **HTTP 000 (unreachable within 15s / down):** sparcoracing.com, rarengineer.com, jparkinsonaz.com,
+  jefffurrier.com — investigate availability separately.
+
+---
+
+## 4. The fix
+
+**Target state per site: `ea-phpXX` (keep the site's current major version) + PHP-FPM + OPcache.**
+
+Two site classes:
+
+1. **suPHP/CGI sites (61)** — just enable FPM. OPcache is already installed+enabled for ea-php81/82/84/74,
+   so it becomes effective the moment a persistent FPM pool exists.
+   `whmapi1 php_set_vhost_versions version=<current ea-phpXX> vhost=<domain> php_fpm=1`
+   (If a site is on ea-php83 or ea-php80 and OPcache pkg is absent for that version, `yum -y install
+   ea-phpXX-php-opcache` first — one-time, per version.)
+
+2. **alt-php84 Selector sites (10)** — reset the Selector to native, then set ea-php84 + FPM (ea-php84
+   OPcache pkg was installed during the pilot):
+   `selectorctl --set-user-current=native --user=<user>` → `php_set_vhost_versions version=ea-php84 vhost=<domain> php_fpm=1`
+
+**Why FPM over "just enable OPcache on the Selector":** FPM is the cPanel-native, supported path; it gives a
+persistent pool (so SHM OPcache actually persists), removes the Selector-vs-MultiPHP ambiguity, and needs no
+per-user extension toggling. It also self-generates the `.user.ini` (php_value/php_flag from `.htaccess` do
+not apply under FPM), which cPanel already maintains.
+
+---
+
+## 5. Rollout plan (phased, verified, reversible)
+
+- **Phase 0 — Pilot: DONE.** ezfastautoglass.com moved to ea-php84 + FPM + OPcache. 3.1s -> 0.44s, bodies clean,
+  Selector reset to native. Auto-rollback armed, never triggered.
+- **Phase 1 — Top offenders (the 5s / 3s tier, ~10 sites):** highest user-visible win. Per-site, with the same
+  execute-with-auto-rollback script (body check on homepage + subpages; revert instantly on any DB/fatal/500).
+- **Phase 2 — Remaining fix candidates (down to ~1s).**
+- **Phase 3 — The fast-but-cold remainder** (low TTFB today, still benefits under load / plugin growth) +
+  decide whether to make **FPM the default for new accounts** (WHM Tweak Settings / a `postwwwacct` hook) so
+  this doesn't silently reaccumulate.
+
+**Per-site procedure (non-negotiable, from the SSD-move incident lessons):**
+1. Pre-check bodies clean (abort if already erroring).
+2. `.htaccess` `php_value`/`php_flag` are cPanel-guarded by `<IfModule>` and do **not** 500 under FPM (mod_lsapi
+   stays loaded and registers the directives; the blocks go inert). Still verify per site — custom `.htaccess`
+   can differ.
+3. Flip FPM (+ Selector native for alt-php sites).
+4. Verify by **response BODY** (not HTTP status) on homepage + 2-3 subpages; confirm `SAPI=fpm-fcgi`,
+   `OPCACHE=YES`, PID stable, TTFB dropped.
+5. On any error marker -> **auto-rollback** to the captured prior state, re-verify clean.
+6. Check the regenerated `.user.ini` `session.save_path` points at the new ea version (cPanel auto-fixes this).
+
+**Capacity:** default cPanel FPM pools are `pm=ondemand` (workers spawn on demand, die when idle), so ~72 pools
+cost little idle memory. Box has 62GB RAM (~52GB free) and 64 cores — ample. Monitor
+`/opt/cpanel/ea-phpXX/root/etc/php-fpm.d/*` pool sizing if any high-traffic site needs `pm=dynamic` tuning.
+
+**OPcache tuning applied (pilot, server-wide for ea-php84):** `memory_consumption=256`,
+`max_accelerated_files=16229` (default 4000 was 99.6% full on a 31-plugin WP — one update from thrashing),
+`interned_strings_buffer=32`. `validate_timestamps` left ON (2s revalidate) so plugin/theme edits go live
+without a manual opcache reset — correct for client WordPress. Apply the same limits to other ea versions as
+sites move onto FPM.
+
+---
+
+## 6. What was changed in the pilot (for the record)
+
+- Installed `ea-php84-php-opcache` (was absent — the reason ea-php84 FPM alone didn't help until installed).
+- `ezfastautoglass.com`: Selector alt-php84 -> native; MultiPHP -> ea-php84 + `php_fpm=1`.
+- Raised ea-php84 OPcache limits (section 5).
+- No client data touched; no DB changes; no `.htaccess` edits (guards made them safe as-is).
+
+**Verification (post):** `SAPI=fpm-fcgi VER=8.4.22 OPCACHE=YES CACHED~4300/16229 MEMfree~111/256MB`; homepage +
+privacy-policy HTTP 200, full byte counts; TTFB 0.41-0.48s across repeated warm hits.
+
+---
+
+## 7. Open items
+
+- Get go/no-go on Phase 1 and the rollout order.
+- Decide on **FPM-as-default for new accounts** (prevents silent reaccumulation).
+- Separately triage the **hightechmortgage.com 500** and the **4 unreachable sites** (availability, unrelated to perf).
+- Install `ea-php83-php-opcache` **iff** any site is moved onto ea-php83.

clients/internal-infrastructure/reports/2026-07-20-ix-web-tier-to-ssd-plan.md

@@ -0,0 +1,49 @@
+# Plan — Web-docroot tiering to SSD on IX (per-user), CageFS-aware
+
+**Date:** 2026-07-20 · **Server:** ix.azcomputerguru.com (CloudLinux 9 + **CageFS** + cPanel/WHM)
+**Status:** PLAN / proposal — not executed. Sequel to the completed DB-datadir SSD move.
+
+## Objective & where we are
+- **Done:** MariaDB datadir is on the SSD RAID1 (the hottest random I/O — the dominant win, ~45× on random reads).
+- **This plan:** move **web docroots** (`public_html`, ~131 GB total across ~accounts) onto the SSD, keeping **mail** (~657 GB, mostly cold) on the HDD `cs-home`. SSD has ~1.45 TB usable, ~16 GB used — ample room.
+
+## Why this is HARDER than the DB move — two structural constraints
+The DB move was one path (`/var/lib/mysql`), server-wide, path-preserving. Per-user web is different:
+
+1. **CageFS** — every account runs in a jailed filesystem; CageFS bind-maps the user's `/home/<user>` into their jail. Anything we do to a docroot must remain visible *inside the jail*. (This is exactly what bit us on the DB socket: root saw it, jails didn't, until `cagefsctl --remount-all`.) A docroot physically on `/ssd` (outside `/home`) is **not in the jail** unless we either (a) keep the *path* under `/home` via a bind mount + propagate/remount into jails, or (b) add `/ssd` to `/etc/cagefs/cagefs.mp` and remount. Either way, **CageFS must be remounted after the change, and it must be tested on ONE account first.**
+
+2. **cPanel disk quotas (the real blocker)** — cPanel enforces and reports per-account disk usage via **filesystem quotas on `cs-home`**. Files physically on the **SSD (a different filesystem)** are **not counted** by cPanel's quota/usage. Result: accounts could silently exceed plan limits, and cPanel disk-usage reporting/billing would be wrong. **cPanel does not officially support account data outside `/home`.** This is a genuine limitation, not a config we can cleanly work around per-account.
+
+## Two approaches
+
+### Approach A — Per-user docroot relocation (what was asked)
+For each account: `mv /home/<u>/public_html /ssd/web/<u>` → recreate `/home/<u>/public_html` → `mount --bind /ssd/web/<u> /home/<u>/public_html` (path stays under `/home`) → fstab entry → `cagefsctl --remount-all` → verify (body check + caged `php -r`).
+- **Pros:** deterministic (you know web is on SSD); mail stays cheap.
+- **Cons / risks:**
+  - **Quota accounting breaks** (see constraint #2) — the big one for a hosting provider.
+  - Nested bind-mount visibility inside CageFS jails needs propagation handling + remount; fragile, must pilot on one account.
+  - **Per-account + ongoing:** every NEW cPanel account needs the same treatment → a `postwwwacct`/hook + a reconciler, forever.
+  - Addon/subdomain docroots often live *under* `public_html` (fine) but some don't — need per-account audit.
+  - Rollback is per-account (unmount bind, move back).
+
+### Approach B — `lvmcache` on `cs-home` (RECOMMENDED to evaluate first)
+Put an SSD **dm-cache** in front of the `cs-home` HDD LV. Hot blocks (frequently-read web *and* mail) migrate to SSD automatically, transparently.
+- **Pros:** **zero per-account work, zero CageFS impact, zero quota impact** (all data stays on `cs-home`, just cached); accelerates web **and** hot mail; one-time setup; no new-account maintenance.
+- **Cons:** LVM surgery on the *live* `/home` volume (do with backups + care); consumer BX500 has no PLP → use **writethrough** cache mode (still gives the read-latency win, which is the point) rather than writeback; and the SSD is currently a VD hosting the DB filesystem, so we'd **carve the SSD** into a DB partition (keep) + a cache LV, or reconsider the DB-on-dedicated-VD vs cache split.
+- **Net:** achieves the same practical goal (web feels fast) without fighting cPanel/CageFS.
+
+## Recommendation
+1. **The dominant win is already captured** (DB on SSD). Web files also benefit heavily from the **OS page cache** (hot files served from the 62 GB RAM regardless of disk), so the marginal value of web-on-SSD is smaller than the DB move was.
+2. Given the **quota + CageFS** constraints and today's incident, I recommend **NOT** doing per-user docroot relocation (Approach A) as the default — it fights cPanel's design and adds permanent per-account fragility.
+3. Instead: **(a) monitor** whether web random-read latency is actually a problem now that the DB is off the HDD (re-run the fio/iostat during peak), and **(b)** if it is, pursue **`lvmcache` (Approach B)** as a carefully-planned, backup-first project — it gets the benefit transparently.
+4. If per-user relocation is still wanted for specific heavy sites, do it as a **narrow, piloted exception** (one account, full CageFS + quota verification) rather than fleet-wide.
+
+## Non-negotiable discipline for whichever path (lessons from today)
+- Pilot on ONE account; verify by **response BODY** (not HTTP status) and by **caged** `sudo -u <user> php -r 'mysqli/opening a file'`, not just root.
+- After any mount/path change: **`cagefsctl --remount-all`** and confirm inside a jail.
+- Keep an instant rollback (original in place) + a backup before touching live data.
+- Test DB/user auth with `mysql --no-defaults -u U -pPASS`, never `MYSQL_PWD`.
+
+## Open questions for Mike
+- Is the driver "web feels slow" (→ measure first, likely `lvmcache`), or a specific hot site (→ narrow pilot)?
+- Are we willing to accept cPanel quota/usage inaccuracy for relocated accounts (Approach A), or is transparent caching (Approach B) the requirement?