16 KiB
Session — security.azcomputerguru.com: live deploy + smoke test (found/fixed prod auth bug)
User
- User: Howard Enos (howard)
- Machine: Howard-Home
- Role: tech
Session Summary
Continued the security-assessment work (same HOWARD-HOME session; see the two companion logs from today). Howard requested: make UI elements slightly larger, test every field for usable info, and a big new multi-tenant client portal + ACG sales view. Did the unambiguous pieces, deferred the portal to a feature request, then (on Howard's "b" greenlight) deployed to the live IX host and functionally tested — which surfaced a production-breaking bug.
Bumped the wizard UI sizing ~10-12% (text, inputs, spacing, wider sheet, bigger buttons) and committed
it. Built app/fieldcheck.cjs and audited every questions.json field: 59 fields / 25 scorable, all 25
emit a usable finding on worst-case answers, no orphan/missing score keys, all findings complete, all 13
requiredControls resolve — every field yields usable info. Captured the portal vision as FR-1 in
FEATURES.md (personas, no-auto-sync guardrail, quote→active workflow, 3 auth options) and asked the
foundational auth decision; Howard chose to defer it to a feature request and focus on testing. Added an
executive summary + top-3 prioritized actions + ACG footer to the export (client = recommendations,
internal = + ACG service).
Deployed to IX (172.16.3.10, cPanel account azcomputerguru, docroot
/home/azcomputerguru/public_html/security, PHP 8.1.34) over SSH: backed up the live files, uploaded to
.new, lint-gated (php -l + JSON validate), then atomic-swapped into place. config.php (live secrets)
was never touched. The public URL was already live behind Cloudflare Access (a public hit returns CF
Access 403), so this was an update, not first-time setup.
Smoke-testing on the server found a production-breaking bug: api.php's allow-list check did a
single strcasecmp($email, ALLOWED_EMAIL) against the WHOLE comma-separated list, so once a 2nd address
(howard@) was added on Jun 19, every API call (lookup/save/load/list/export) returned forbidden — the
live backend had been unusable. Fixed it to split + membership-check (matching index.php), redeployed,
and re-verified: action=list reads the DB; internal export renders posture/exec-summary/findings/ACG-
service/REQUIRED/captured-intake; client export renders posture+recommendations with zero upsell leaks
and no raw intake. Inserted a weak-answer test row, exported both views, then deleted it — prod DB back
to its 2 real rows. Committed everything to submodule main and advanced the claudetools pin so git
matches the live server.
Key Decisions
- Deploy via root SSH + base64 file upload, lint-gated atomic swap. Backup → upload
.new→php -l/JSON-validate →mvinto place. Never overwriteconfig.php. Safe for a live site. - Functional-test via PHP CLI + a shipped harness, not inline
php -r. Inlinessh "php -r '...'"mangled through nested quoting (printed PHP usage); a base64'd.phpharness that includes api.php and drives$_GET[action]is reliable. Bypasses the Apache/Cloudflare vhost routing (origin-direct curl 404'd — a vhost/SNI artifact, not a broken deploy). - Test row written to the tool's own MySQL then deleted — acceptable for a smoke test (its own DB,
not Syncro/RMM), clearly tagged
ZZ-SMOKE-DELETE, cleaned up. - Portal deferred to FR-1 (Howard's call) rather than building a multi-tenant auth model speculatively.
- MSYS
sshfor password+askpass, not native Windows OpenSSH (which fails the .sh askpass) — matches the working pfsense-ssh.sh pattern.
Problems Encountered
- PROD BUG — api.php 403 for everyone (single strcasecmp vs comma list). Found via live smoke test; fixed + redeployed + verified. This is why "test every function" mattered.
- Native Windows OpenSSH askpass failed
CreateProcessW error:193(can't exec a .sh askpass) → used MSYS baressh. Logged --friction. - Inline
php -rover ssh mangled (nested quote stripping) → shipped a base64'd .php harness. Logged --friction (reffeedback_windows_quote_stripping). - Origin-direct curl 404'd (vhost/SNI; CF proxies the real path) → tested via PHP CLI instead.
- Submodule reset to old scaffold earlier (stale gitlink + a concurrent submodule-update) → restored to origin/main and advanced the claudetools pin to stop the churn.
Configuration Changes
security-assessment submodule (all on main, pushed; claudetools pin advanced each time):
app/index.php— UI sizing bump (66eb7cb).app/api.php— export exec-summary/top-3/footer (3e3a9ab); allow-list auth fix (f246091).FEATURES.md(new) — FR-1 portal request;app/fieldcheck.cjs(new) — field-audit dev tool (3a2301b).- Local scratch
app/_deploytest.php+app/_smoke.phpcreated for testing, removed after (not committed). Live IX server:index.php,api.php,questions.jsonupdated in the docroot (backups.bak-20260621-181744);config.phpuntouched. ClaudeTools: this session log +errorlog.md(2 friction entries).
Credentials & Secrets
None created/discovered. Used existing vault infrastructure/ix-server (root SSH 172.16.3.10:22, password
field credentials.password; also a WHM API token credentials.whm-api-token = full-access root). DB creds
live in the server's config.php (vault msp-tools/security-assessment-db). No secrets printed/committed.
Infrastructure & Servers
- IX server 172.16.3.10 (ext 72.194.62.5), Rocky Linux WHM/cPanel, PHP 8.1.34 (ea-php81), root SSH :22.
- Site docroot:
/home/azcomputerguru/public_html/security(cPanel acctazcomputerguru, sub of azcomputerguru.com, vhost 172.16.3.10:80/:443). - security.azcomputerguru.com behind Cloudflare Access (Zero Trust app
8ce5f31c-...; allow mike@ + howard@). Origin answers via CF; origin-direct curl needs--resolve/Host and still 404'd (CF-proxied path is the working one). - Live DB (acgsec_assess): 2 real rows (id 1 Darrell Delphen, id 2 empty) after test cleanup.
Commands & Outputs
# find docroot
ssh root@172.16.3.10 'grep -i security.azcomputerguru.com /etc/userdatadomains'
# -> azcomputerguru==root==sub==azcomputerguru.com==/home/azcomputerguru/public_html/security==...ea-php81
# deploy pattern: backup -> base64 upload to .new -> php -l -> chown user -> mv (atomic)
base64 local.php | ssh root@ix "base64 -d > docroot/file.new"
# functional test (bypasses Apache/CF): shipped harness includes api.php
php _deploytest.php list # {"items":[{id:2...},{id:1,...}]} (after the auth fix)
php _deploytest.php export <id> internal # posture/exec/findings/ACG service/Captured intake
php _deploytest.php export <id> client # posture/recommendations, 0 ACG-service, no Captured intake
# BUG before fix: php _deploytest.php list -> {"error":"forbidden"}
Pending / Incomplete Tasks
- Live browser UI click-through (through Cloudflare Access) — the only thing not tested; needs a human logged into https://security.azcomputerguru.com (look up a client, answer fields, watch posture, Export both views). Backend fully verified server-side.
- #1 GuruRMM endpoint prefill — deferred (infra: no Syncro→RMM mapping, no reachable RMM API from IX).
- FR-1 multi-tenant portal — filed in FEATURES.md, awaiting the auth decision when Howard wants it.
Reference Information
- Submodule
mainHEADf246091(claudetools pin27c1d97at time of fix). - Commits this turn:
66eb7cbsizing,3a2301bFR-1+fieldcheck,3e3a9abexport exec-summary,f246091api auth fix. - Live server backups:
*.bak-20260621-181744in the docroot (rollback if needed). - Export endpoint:
api.php?action=export&id=<id>&view=internal|client(origin requires the CF Access email header). - Companion logs:
2026-06-21-howard-security-assessment-scoring.md,...-unifi-pfsense-control-verbs.md,...-gururmm-bug-018-019.md.
Update: 18:53 PT — site was broken in browser (stale opcache); root-caused + fixed; CONFIRMED loading
After the deploy, Howard reported the live page "mostly broken — only the Export and Assessments buttons at the top." Diagnosed and resolved.
What was wrong (two compounding issues)
- api.php allow-list bug (already fixed earlier this session): single
strcasecmp($email, ALLOWED_EMAIL)vs the whole comma string → 403 for every API call. Confirmed live via the access log (POST /api.php?action=save 403repeatedly). - Stale PHP OPcache — the server kept executing the OLD
index.php(rendered output 14637 bytes) and OLDapi.phpeven though the new files were on disk (md5 matched local). So the new wizard never rendered (boot() ran the old code / the old api 403'd) and the fix wasn't live.
Diagnosis path
- Verified deployed files = local (identical md5), perms 644,
.htaccessbenign, questions.json valid on server, no PHP error log → not a crash. Ran the wizard JS in node with DOM/fetch stubs → boot() completed fine (so not a code bug). The Apache SSL access log (/etc/apache2/logs/domlogs/security.azcomputerguru.com-ssl_log) was the smoking gun:GET / 200 14637(old size; new index.php renders ~27.9k) + manyapi.php ... 403. - SAPI = Apache (httpd) + opcache On. Origin-direct curl to 127.0.0.1 hits a default vhost that 404s — the app is only reachable via the EXTERNAL IP path Cloudflare uses.
Fix
- Dropped a one-off
_oc.php(opcache_reset()) in the docroot and ran it through the real path:curl -sk --resolve security.azcomputerguru.com:443:72.194.62.5 -H 'Cf-Access-Authenticated-User-Email: mike@azcomputerguru.com' https://.../_oc.php→opcache_reset: OK(validate_timestamps=1, but the cache hadn't revalidated; reset forced it). - Verified:
GET /now 200 27885 bytes, served HTML containscomputeScores/gradeFor/"Posture & findings";api.php?action=listreturns JSON (no 403). Removed_oc.php+ local scratch (_jsrun.cjs,_deploytest.php,_smoke.php). - Howard confirmed: "it is loading now."
Durable fixes
DEPLOY.mdupdated (31bc786): "FLUSH OPCACHE after updating" (the external-IP_oc.phprecipe) + the comma-separatedALLOWED_EMAILgotcha. claudetools pin advanced (5299f3d). Logged --friction (deploy/cpanel: stale opcache).
Key gotchas for next deploy
- After uploading PHP changes to this cPanel host, opcache can serve stale bytecode → flush it (recipe in DEPLOY.md).
- Origin-direct (127.0.0.1) 404s the vhost; test/flush via
--resolve ...:443:72.194.62.5(external IP). - Coord (Mike, GURU-5070): BUG-018 (cea87d4) + Event Log Watch UI merged to gururmm main + deploying; I'm cleared to self-merge gururmm going forward.
Net state
security.azcomputerguru.com is LIVE and working (new scoring wizard + larger UI + dual export + fixed backend), verified server-side and confirmed loading in Howard's browser. Remaining: #1 RMM prefill (deferred, infra), FR-1 portal (deferred, auth decision).
Update: 19:31 PT — post-deploy live fixes (wrong-customer lookup bug + New/clear button)
Two issues Howard found while using the live site, both fixed + deployed + verified.
Bug: phone lookup returned the WRONG customer
- Symptom: entering
5205851310pulled up "Marjolaine Deslauriers" (an unrelated record). - Root cause: Syncro's
GET /customers?phone=<n>param does NOT filter — it returns the full customer list (5079 across 51 pages); the code tookcustomers[0](Marjolaine, phone 5203310273). Reproduced via the Syncro API (vaultmsp-tools/syncro-howardcred fieldcredentials.credential):?phone=→ count 100 unfiltered;?query=5205851310→ count 0 (number not in Syncro);?query=5208880000(Insty-Prints' real number) → count 1 (so?query=DOES search phone by digits). - Fix (api.php lookup): use
?query=<digits>and VERIFY a candidate's phone digit-tail matches the entered digits before prefilling (handles leading-1 / 7-vs-10 digits). No match → return a clear "No customer matches that phone number" + suggestions, and prefill NOTHING. index.php doLookup updated to show the message/suggestions and not prefill on no-match. Format was never the issue (input is digit-normalized, so 5205851310 == (520) 585-1310). - Verified live (web POST via the external-IP path):
5205851310→ no-match;5208880000→ Insty-Prints full prefill;(520) 585-1310→ no-match (same as digits).
Feature: "+ New" button (Howard's request)
- After loading a client,
state.idpersisted, so a fresh lookup + save could overwrite the previous client's record. Added a top-bar + New button ->clearAll()resets id/data/consent/sec/view + URL + topbar, so the next save INSERTs a new record. Confirms before clearing if work is in progress.
Deploy mechanics (each fix)
- Upload changed file(s) to
.new->php -l-> atomicmv-> flush opcache via a one-off_oc.phphit through--resolve ...:443:72.194.62.5(external IP; 127.0.0.1 vhost 404s) -> verify served HTML. Backups*.bak-<ts>in the docroot. Browser needs Ctrl+Shift+R to pick up index.php changes.
Commits
ef03b4elookup phone-match fix;a9c85a7+ New button. claudetools pins advanced (c462f50,5b44a3e).
Net
Phone search works correctly (verified by Howard), wrong-customer risk eliminated, and the + New button prevents cross-record contamination. Site fully functional.
Update: 21:25 PT — onboarding/upsell questions + opportunity engine + Save-quote + quote delete
Expanded the assessment per Howard ("what other questions to better integrate clients + gather upsell data") plus quote lifecycle controls. All deployed to IX + verified live.
Save-quote + status
- Added a top-bar Save quote button + QUOTE badge; added a
statuscolumn (defaultquote) to theassessmentstable (ALTER on live DB + schema.sql). New saves arequote(unverified, never auto-synced); UPDATE leaves status (preserves a future promotion).listreturns status; Assessments list shows it.
New questions (Howard's "go with recommendations")
- Security-scored: M365 license tier (Basic/Standard -> upgrade to Business Premium; it unlocks MFA/CA/Intune/Defender).
- New Other Services section: VoIP (PacketDial), managed print, surveillance, web hosting.
- New Onboarding & Sales Context section (internal-only): current support model, existing agents to remove, admin-access readiness, hardware age, cyber-insurance renewal timing, decision-maker, biggest frustration, planned changes.
- Plus WiFi coverage, internet failover, UPS.
Opportunity engine (the key design)
- New field-level
opportunityblock (parallel torisk): non-security upsells that fire on awhentrigger ({equals|in|includes|notIncludes}) and surface for SALES WITHOUT affecting the security grade. - Engine mirrored JS (
computeScoresreturnsopportunities) + PHP (score_assessment+opp_fires). - Export: INTERNAL shows an "Upsell opportunities" section (each -> ACG service); CLIENT view hides them entirely. Wizard internal view shows them; client toggle hides. Verified: 6 opportunities fire on a weak sample, security findings stay separate (2).
Quote lifecycle
- api
promote(quote->active) +deleteactions (ACG-gated). Assessments list: Delete works (confirm + clears the loaded record if it's the one deleted); Activate/make-live button DISABLED per Howard (live import to Syncro/RMM stays manual until wired; promote API is dormant/ready).
Problem hit
- First deploy shipped api.php + index.php but NOT the new questions.json -> opportunities/new questions didn't appear live (OPPS=0). Root-caused via a CLI opp-debug (server questions.json was old). Deployed questions.json (lint-gated) -> verified opportunities fire + new sections live. Lesson: deploy ALL changed files, incl. data.
Commits / state
- Submodule
0f6927b(questions+engine+delete) on main; earlier this session66eb7cb/3a2301b/3e3a9ab/f246091/a9c85a7/7a3dfb8. claudetools pin0f6927b(HEAD8713b46). All live on IX, opcache flushed. - Concurrent screenconnect WIP in the shared tree preserved through the pin rebases.
Still open
- Activate/promote UI intentionally disabled (wire live import later). FR-1 portal still deferred (auth decision).
- #1 RMM prefill deferred (infra). Optional: extend fieldcheck.cjs to validate
opportunityfields.