diff --git a/.claude/memory/MEMORY.md b/.claude/memory/MEMORY.md index 97a813a4..6ce4adbd 100644 --- a/.claude/memory/MEMORY.md +++ b/.claude/memory/MEMORY.md @@ -26,6 +26,7 @@ - [Community Forum (Flarum)](reference_community_forum.md) — Flarum forum at community.azcomputerguru.com, API access, database, posting workflow. - [Radio Show Website](reference_radio_website.md) — Astro static site at radio.azcomputerguru.com on IX server. - [IX PHP-FPM/OPcache remediation](project_ix_php_fpm_opcache_remediation.md) — IX server default is suPHP/CGI per-request with OPcache cold: 71/72 sites recompile PHP every request (1-5.6s TTFB). Fix = per-vhost PHP-FPM. ezfast pilot done (3.1s->0.44s); fleet rollout proposed. +- [Profile before hardware](feedback_profile_before_hardware.md) — For ANY performance complaint, measure/decompose the actual bottleneck BEFORE proposing hardware; config/cheapest cause first. (IX: $600 SSDs recommended before finding the real cause was OPcache-off — a config issue.) - [IX Server Access](reference_ix_server_access.md) — `ix.azcomputerguru.com` / 172.16.3.10. Reachable when Tailscale is on (no VPN). SSH currently uses sshpass with root password; key auth from GURU-5070 not configured yet (was CachyOS, now Win11 — verify). - [Cloudflare access](reference_cloudflare_access.md) — Cloudflare API creds in SOPS `services/cloudflare.sops.yaml` (full DNS + account tokens; azcomputerguru zone_id 1beb9917...). azcomputerguru.com DNS is on Cloudflare (not IX) — edit via Cloudflare API, not whmapi1. - [Matomo Analytics](reference_matomo_analytics.md) — Self-hosted analytics at analytics.azcomputerguru.com, site IDs, tracking for all 3 sites. diff --git a/.claude/memory/feedback_profile_before_hardware.md b/.claude/memory/feedback_profile_before_hardware.md new file mode 100644 index 00000000..fa7d643a --- /dev/null +++ b/.claude/memory/feedback_profile_before_hardware.md @@ -0,0 +1,31 @@ +--- +name: feedback_profile_before_hardware +description: For ANY performance complaint, profile the actual bottleneck and decompose latency BEFORE proposing/buying hardware — config causes first. +metadata: + type: feedback +--- + +For any "it's slow" / performance concern, **measure where the time actually goes before proposing +hardware or any spend.** Decompose the symptom the user cares about (page load = TTFB breakdown; +"server slow" = which resource), find the dominant cost, then fix the cheapest cause first (usually +config), and only reach for hardware when the numbers demand it. + +**Why:** On IX (2026-07), Mike asked about server performance several times. Each time I answered the +narrow framing — benchmarked storage I/O, checked RAID, sized and recommended **$600** of SSDs+trays — +without ever profiling the request path. The real dominant cause of slow page loads was **PHP +per-request compile with OPcache off** (a fleet-wide config default), which a 30-second test (trivial +`.php` returns 0.13s vs WP 3s) would have shown day one. He was rightly disappointed: "I count on you +to get things right the first time, not to stumble upon the answer later." The drives do help DB random +I/O (real, ~45x), but buying them before ruling out config inverted the correct order of work. + +**How to apply:** +- "Site/page is slow" -> FIRST measure TTFB and split it: DNS/connect/TLS vs server think-time. If + think-time dominates, it's server-side PHP/DB, not the network/disk. Then profile PHP: a trivial + `.php` vs the real page isolates compile/bootstrap; check the SAPI + OPcache (`php_sapi_name()`, + `opcache_get_status()`, PID-stability across requests) BEFORE touching disk/hardware. +- "Server is slow" -> check load, per-resource saturation (CPU/RAM/IO), and the actual hot path before + recommending a purchase. Name the measured bottleneck and its number. +- Do NOT let the user's framing ("benchmark storage") narrow the diagnosis past the symptom they care + about. Answer the framing, but ALSO measure the real end-to-end bottleneck. +- This is a concrete instance of the CLAUDE.md **"Simplest solution FIRST"** rule: config/measurement + before hardware. See [[project_ix_php_fpm_opcache_remediation]] for the fix that resulted. diff --git a/clients/internal-infrastructure/reports/2026-07-20-ix-php-fpm-opcache-remediation.md b/clients/internal-infrastructure/reports/2026-07-20-ix-php-fpm-opcache-remediation.md index cd160120..e66eebb0 100644 --- a/clients/internal-infrastructure/reports/2026-07-20-ix-php-fpm-opcache-remediation.md +++ b/clients/internal-infrastructure/reports/2026-07-20-ix-php-fpm-opcache-remediation.md @@ -1,7 +1,7 @@ # Remediation — IX PHP handler / OPcache (fleet-wide) + ezfastautoglass pilot **Date:** 2026-07-20 · **Server:** ix.azcomputerguru.com (CloudLinux 9 + CageFS + cPanel/WHM, Dell R740xd) -**Author:** Mike Swanson (GURU-5070) · **Status:** Pilot COMPLETE (ezfastautoglass), fleet rollout PROPOSED (awaiting go) +**Author:** Mike Swanson (GURU-5070) · **Status:** Pilot + Phase 1 COMPLETE (12 sites); Phases 2-3 PROPOSED (awaiting go) **Related:** `2026-07-20-ix-mysql-ssd-move-incident-postmortem.md`, `2026-07-20-ix-web-tier-to-ssd-plan.md` --- @@ -115,9 +115,13 @@ not apply under FPM), which cPanel already maintains. - **Phase 0 — Pilot: DONE.** ezfastautoglass.com moved to ea-php84 + FPM + OPcache. 3.1s -> 0.44s, bodies clean, Selector reset to native. Auto-rollback armed, never triggered. -- **Phase 1 — Top offenders (the 5s / 3s tier, ~10 sites):** highest user-visible win. Per-site, with the same - execute-with-auto-rollback script (body check on homepage + subpages; revert instantly on any DB/fatal/500). -- **Phase 2 — Remaining fix candidates (down to ~1s).** +- **Phase 1 — Top 10 offenders: DONE (2026-07-20).** All flipped to their current ea-phpXX + FPM (no version + change), OPcache confirmed effective, bodies clean, zero rollbacks. Before -> after TTFB: + bestmassageintucson 6.72->1.18 · jrkco 5.89->1.17 · tucsonadobe 4.53->0.54 · sandtekomachinery 3.68->0.81 · + arizonahatters 3.18->0.79 · fsgtucson 3.32->0.50 · acepickupparts 3.36->0.62 · gurushow 2.35->0.60 · + tarafulcreations 2.85->0.55 · phxpropane 2.76->0.37. (ea-php80/82 opcache pkgs already present; ea-php84 + installed during pilot.) +- **Phase 2 — Remaining fix candidates (down to ~1s), ~26 sites, incl. the 10 alt-php Selector sites.** - **Phase 3 — The fast-but-cold remainder** (low TTFB today, still benefits under load / plugin growth) + decide whether to make **FPM the default for new accounts** (WHM Tweak Settings / a `postwwwacct` hook) so this doesn't silently reaccumulate. @@ -159,7 +163,11 @@ privacy-policy HTTP 200, full byte counts; TTFB 0.41-0.48s across repeated warm ## 7. Open items -- Get go/no-go on Phase 1 and the rollout order. +- Get go/no-go on **Phase 2** (remaining ~26 sites incl. the 10 alt-php Selector sites) and rollout order. - Decide on **FPM-as-default for new accounts** (prevents silent reaccumulation). -- Separately triage the **hightechmortgage.com 500** and the **4 unreachable sites** (availability, unrelated to perf). +- **hightechmortgage.com 500: RESOLVED (2026-07-20)** — root cause was PHP OOM (`memory_limit` 128M exhausted + by WooCommerce + AccelerateWP + crypto plugins), not the handler. Raised to 512M via `.user.ini`; site + returns 200, no new fatals. Will move to FPM in Phase 2 (memory_limit carries over). +- Separately triage the **4 unreachable sites** (HTTP 000: sparcoracing, rarengineer, jparkinsonaz, + jefffurrier) — availability, unrelated to perf. - Install `ea-php83-php-opcache` **iff** any site is moved onto ea-php83. diff --git a/errorlog.md b/errorlog.md index 36467fe9..4b5a920c 100644 --- a/errorlog.md +++ b/errorlog.md @@ -19,6 +19,10 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure · +2026-07-21 | GURU-5070 | performance-diagnosis | [correction] assumed 'slow IX server' = storage I/O; benchmarked disks/RAID and recommended $600 in SSDs+trays before ever profiling the request path. Actual dominant cause of slow page loads was PHP per-request compile with OPcache off (a fleet-wide CONFIG issue). A 30-second test (trivial .php 0.13s vs WP 3s) would have shown it day one. Rule: for any performance complaint, PROFILE the actual bottleneck and decompose latency BEFORE proposing hardware; cheapest/config cause first (ties to 'Simplest solution FIRST'). + +2026-07-21 | GURU-5070 | mailprotector | mp.py messages domains raises unhandled traceback; logs/messages at domain scope return empty despite active mail [ctx: domain=27668 cmd=messages/logs] + 2026-07-21 | Howard-Home | ps-encoded | EncodedCommand dispatch failed [ctx: agent=fa99e913-1027-4e33-a928-7695e31068e7 resp=Agent not found] 2026-07-20 | Howard-Home | harness/skill-routing | [correction] Skills being bypassed -- hand-rolling API calls when skills exist (email/mailbox, unifi, others). CLAUDE.md skill-first rule not sufficient as a guardrail. [ctx: ref=CLAUDE.md-skill-first-rule] @@ -34,7 +38,7 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure · 2026-07-20 | Howard-Home | remediation-tool | get-token: token request failed (no access_token) [ctx: tenant=207fa277-e9d8-4eb7-ada1-1064d2221498 tier=investigator auth=secret err=7000215] 2026-07-21 | GURU-5070 | cloudlinux/cagefs | [correction] After moving the MySQL datadir (or changing its socket) on a CloudLinux CageFS server, you MUST run 'cagefsctl --remount-all' — otherwise caged web users get 'mysqli_connect(): (HY000/2002): No such file or directory' even though root + the raw socket are fine (root isn't jailed, so root-shell php connects but the sites don't). This was the REAL cause of the IX site outages during the MySQL SSD datadir move; missing it sent me down a phantom auth rabbit hole. [ctx: host=ix ref=ix-mysql-ssd-move-postmortem] -2026-07-21 | GURU-5070 | grok | grok returned no text [ctx: mode=text stopReason=] (x2) +2026-07-21 | GURU-5070 | grok | grok returned no text [ctx: mode=text stopReason=] (x3) 2026-07-21 | GURU-5070 | bash/mysql-testing | [friction] Wasted a LONG production incident chasing a phantom 'grant management broken' MariaDB bug (blamed the datadir move, Aria corruption, auth_socket, governor) — root cause was MY test method: /root/.my.cnf has a password, which OVERRIDES MYSQL_PWD env, so 'MYSQL_PWD=x mysql -u testuser' silently sent ROOT's password with the test username -> always Access denied. ALWAYS use 'mysql --no-defaults -u USER -pPASS' (or an explicit --defaults-file) when testing a NON-root user's auth. Caused multiple unnecessary prod restarts + a datadir rollback. [ctx: host=ix ref=my.cnf-overrides-MYSQL_PWD]