IX: Phase 1 done (10 sites -> FPM/OPcache), hightechmortgage 500 fixed + profile-first lesson
Phase 1 top-10 offenders migrated to ea-phpXX + PHP-FPM + OPcache, all verified clean, zero rollbacks (3-7s -> sub-1s TTFB). hightechmortgage.com 500 resolved (OOM: memory_limit 128M->512M, not a handler issue). Logged the diagnosis correction (profile the bottleneck before recommending hardware) as errorlog --correction + feedback memory. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -26,6 +26,7 @@
|
||||
- [Community Forum (Flarum)](reference_community_forum.md) — Flarum forum at community.azcomputerguru.com, API access, database, posting workflow.
|
||||
- [Radio Show Website](reference_radio_website.md) — Astro static site at radio.azcomputerguru.com on IX server.
|
||||
- [IX PHP-FPM/OPcache remediation](project_ix_php_fpm_opcache_remediation.md) — IX server default is suPHP/CGI per-request with OPcache cold: 71/72 sites recompile PHP every request (1-5.6s TTFB). Fix = per-vhost PHP-FPM. ezfast pilot done (3.1s->0.44s); fleet rollout proposed.
|
||||
- [Profile before hardware](feedback_profile_before_hardware.md) — For ANY performance complaint, measure/decompose the actual bottleneck BEFORE proposing hardware; config/cheapest cause first. (IX: $600 SSDs recommended before finding the real cause was OPcache-off — a config issue.)
|
||||
- [IX Server Access](reference_ix_server_access.md) — `ix.azcomputerguru.com` / 172.16.3.10. Reachable when Tailscale is on (no VPN). SSH currently uses sshpass with root password; key auth from GURU-5070 not configured yet (was CachyOS, now Win11 — verify).
|
||||
- [Cloudflare access](reference_cloudflare_access.md) — Cloudflare API creds in SOPS `services/cloudflare.sops.yaml` (full DNS + account tokens; azcomputerguru zone_id 1beb9917...). azcomputerguru.com DNS is on Cloudflare (not IX) — edit via Cloudflare API, not whmapi1.
|
||||
- [Matomo Analytics](reference_matomo_analytics.md) — Self-hosted analytics at analytics.azcomputerguru.com, site IDs, tracking for all 3 sites.
|
||||
|
||||
31
.claude/memory/feedback_profile_before_hardware.md
Normal file
31
.claude/memory/feedback_profile_before_hardware.md
Normal file
@@ -0,0 +1,31 @@
|
||||
---
|
||||
name: feedback_profile_before_hardware
|
||||
description: For ANY performance complaint, profile the actual bottleneck and decompose latency BEFORE proposing/buying hardware — config causes first.
|
||||
metadata:
|
||||
type: feedback
|
||||
---
|
||||
|
||||
For any "it's slow" / performance concern, **measure where the time actually goes before proposing
|
||||
hardware or any spend.** Decompose the symptom the user cares about (page load = TTFB breakdown;
|
||||
"server slow" = which resource), find the dominant cost, then fix the cheapest cause first (usually
|
||||
config), and only reach for hardware when the numbers demand it.
|
||||
|
||||
**Why:** On IX (2026-07), Mike asked about server performance several times. Each time I answered the
|
||||
narrow framing — benchmarked storage I/O, checked RAID, sized and recommended **$600** of SSDs+trays —
|
||||
without ever profiling the request path. The real dominant cause of slow page loads was **PHP
|
||||
per-request compile with OPcache off** (a fleet-wide config default), which a 30-second test (trivial
|
||||
`.php` returns 0.13s vs WP 3s) would have shown day one. He was rightly disappointed: "I count on you
|
||||
to get things right the first time, not to stumble upon the answer later." The drives do help DB random
|
||||
I/O (real, ~45x), but buying them before ruling out config inverted the correct order of work.
|
||||
|
||||
**How to apply:**
|
||||
- "Site/page is slow" -> FIRST measure TTFB and split it: DNS/connect/TLS vs server think-time. If
|
||||
think-time dominates, it's server-side PHP/DB, not the network/disk. Then profile PHP: a trivial
|
||||
`.php` vs the real page isolates compile/bootstrap; check the SAPI + OPcache (`php_sapi_name()`,
|
||||
`opcache_get_status()`, PID-stability across requests) BEFORE touching disk/hardware.
|
||||
- "Server is slow" -> check load, per-resource saturation (CPU/RAM/IO), and the actual hot path before
|
||||
recommending a purchase. Name the measured bottleneck and its number.
|
||||
- Do NOT let the user's framing ("benchmark storage") narrow the diagnosis past the symptom they care
|
||||
about. Answer the framing, but ALSO measure the real end-to-end bottleneck.
|
||||
- This is a concrete instance of the CLAUDE.md **"Simplest solution FIRST"** rule: config/measurement
|
||||
before hardware. See [[project_ix_php_fpm_opcache_remediation]] for the fix that resulted.
|
||||
@@ -1,7 +1,7 @@
|
||||
# Remediation — IX PHP handler / OPcache (fleet-wide) + ezfastautoglass pilot
|
||||
|
||||
**Date:** 2026-07-20 · **Server:** ix.azcomputerguru.com (CloudLinux 9 + CageFS + cPanel/WHM, Dell R740xd)
|
||||
**Author:** Mike Swanson (GURU-5070) · **Status:** Pilot COMPLETE (ezfastautoglass), fleet rollout PROPOSED (awaiting go)
|
||||
**Author:** Mike Swanson (GURU-5070) · **Status:** Pilot + Phase 1 COMPLETE (12 sites); Phases 2-3 PROPOSED (awaiting go)
|
||||
**Related:** `2026-07-20-ix-mysql-ssd-move-incident-postmortem.md`, `2026-07-20-ix-web-tier-to-ssd-plan.md`
|
||||
|
||||
---
|
||||
@@ -115,9 +115,13 @@ not apply under FPM), which cPanel already maintains.
|
||||
|
||||
- **Phase 0 — Pilot: DONE.** ezfastautoglass.com moved to ea-php84 + FPM + OPcache. 3.1s -> 0.44s, bodies clean,
|
||||
Selector reset to native. Auto-rollback armed, never triggered.
|
||||
- **Phase 1 — Top offenders (the 5s / 3s tier, ~10 sites):** highest user-visible win. Per-site, with the same
|
||||
execute-with-auto-rollback script (body check on homepage + subpages; revert instantly on any DB/fatal/500).
|
||||
- **Phase 2 — Remaining fix candidates (down to ~1s).**
|
||||
- **Phase 1 — Top 10 offenders: DONE (2026-07-20).** All flipped to their current ea-phpXX + FPM (no version
|
||||
change), OPcache confirmed effective, bodies clean, zero rollbacks. Before -> after TTFB:
|
||||
bestmassageintucson 6.72->1.18 · jrkco 5.89->1.17 · tucsonadobe 4.53->0.54 · sandtekomachinery 3.68->0.81 ·
|
||||
arizonahatters 3.18->0.79 · fsgtucson 3.32->0.50 · acepickupparts 3.36->0.62 · gurushow 2.35->0.60 ·
|
||||
tarafulcreations 2.85->0.55 · phxpropane 2.76->0.37. (ea-php80/82 opcache pkgs already present; ea-php84
|
||||
installed during pilot.)
|
||||
- **Phase 2 — Remaining fix candidates (down to ~1s), ~26 sites, incl. the 10 alt-php Selector sites.**
|
||||
- **Phase 3 — The fast-but-cold remainder** (low TTFB today, still benefits under load / plugin growth) +
|
||||
decide whether to make **FPM the default for new accounts** (WHM Tweak Settings / a `postwwwacct` hook) so
|
||||
this doesn't silently reaccumulate.
|
||||
@@ -159,7 +163,11 @@ privacy-policy HTTP 200, full byte counts; TTFB 0.41-0.48s across repeated warm
|
||||
|
||||
## 7. Open items
|
||||
|
||||
- Get go/no-go on Phase 1 and the rollout order.
|
||||
- Get go/no-go on **Phase 2** (remaining ~26 sites incl. the 10 alt-php Selector sites) and rollout order.
|
||||
- Decide on **FPM-as-default for new accounts** (prevents silent reaccumulation).
|
||||
- Separately triage the **hightechmortgage.com 500** and the **4 unreachable sites** (availability, unrelated to perf).
|
||||
- **hightechmortgage.com 500: RESOLVED (2026-07-20)** — root cause was PHP OOM (`memory_limit` 128M exhausted
|
||||
by WooCommerce + AccelerateWP + crypto plugins), not the handler. Raised to 512M via `.user.ini`; site
|
||||
returns 200, no new fatals. Will move to FPM in Phase 2 (memory_limit carries over).
|
||||
- Separately triage the **4 unreachable sites** (HTTP 000: sparcoracing, rarengineer, jparkinsonaz,
|
||||
jefffurrier) — availability, unrelated to perf.
|
||||
- Install `ea-php83-php-opcache` **iff** any site is moved onto ea-php83.
|
||||
|
||||
@@ -19,6 +19,10 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure ·
|
||||
|
||||
<!-- Append entries below this line -->
|
||||
|
||||
2026-07-21 | GURU-5070 | performance-diagnosis | [correction] assumed 'slow IX server' = storage I/O; benchmarked disks/RAID and recommended $600 in SSDs+trays before ever profiling the request path. Actual dominant cause of slow page loads was PHP per-request compile with OPcache off (a fleet-wide CONFIG issue). A 30-second test (trivial .php 0.13s vs WP 3s) would have shown it day one. Rule: for any performance complaint, PROFILE the actual bottleneck and decompose latency BEFORE proposing hardware; cheapest/config cause first (ties to 'Simplest solution FIRST').
|
||||
|
||||
2026-07-21 | GURU-5070 | mailprotector | mp.py messages domains <id> raises unhandled traceback; logs/messages at domain scope return empty despite active mail [ctx: domain=27668 cmd=messages/logs]
|
||||
|
||||
2026-07-21 | Howard-Home | ps-encoded | EncodedCommand dispatch failed [ctx: agent=fa99e913-1027-4e33-a928-7695e31068e7 resp=Agent not found]
|
||||
|
||||
2026-07-20 | Howard-Home | harness/skill-routing | [correction] Skills being bypassed -- hand-rolling API calls when skills exist (email/mailbox, unifi, others). CLAUDE.md skill-first rule not sufficient as a guardrail. [ctx: ref=CLAUDE.md-skill-first-rule]
|
||||
@@ -34,7 +38,7 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure ·
|
||||
2026-07-20 | Howard-Home | remediation-tool | get-token: token request failed (no access_token) [ctx: tenant=207fa277-e9d8-4eb7-ada1-1064d2221498 tier=investigator auth=secret err=7000215]
|
||||
2026-07-21 | GURU-5070 | cloudlinux/cagefs | [correction] After moving the MySQL datadir (or changing its socket) on a CloudLinux CageFS server, you MUST run 'cagefsctl --remount-all' — otherwise caged web users get 'mysqli_connect(): (HY000/2002): No such file or directory' even though root + the raw socket are fine (root isn't jailed, so root-shell php connects but the sites don't). This was the REAL cause of the IX site outages during the MySQL SSD datadir move; missing it sent me down a phantom auth rabbit hole. [ctx: host=ix ref=ix-mysql-ssd-move-postmortem]
|
||||
|
||||
2026-07-21 | GURU-5070 | grok | grok returned no text [ctx: mode=text stopReason=] (x2)
|
||||
2026-07-21 | GURU-5070 | grok | grok returned no text [ctx: mode=text stopReason=] (x3)
|
||||
|
||||
2026-07-21 | GURU-5070 | bash/mysql-testing | [friction] Wasted a LONG production incident chasing a phantom 'grant management broken' MariaDB bug (blamed the datadir move, Aria corruption, auth_socket, governor) — root cause was MY test method: /root/.my.cnf has a password, which OVERRIDES MYSQL_PWD env, so 'MYSQL_PWD=x mysql -u testuser' silently sent ROOT's password with the test username -> always Access denied. ALWAYS use 'mysql --no-defaults -u USER -pPASS' (or an explicit --defaults-file) when testing a NON-root user's auth. Caused multiple unnecessary prod restarts + a datadir rollback. [ctx: host=ix ref=my.cnf-overrides-MYSQL_PWD]
|
||||
|
||||
|
||||
Reference in New Issue
Block a user