# Session — security.azcomputerguru.com: live deploy + smoke test (found/fixed prod auth bug)

## User
- **User:** Howard Enos (howard)
- **Machine:** Howard-Home
- **Role:** tech

## Session Summary

Continued the security-assessment work (same HOWARD-HOME session; see the two companion logs from
today). Howard requested: make UI elements slightly larger, test every field for usable info, and a
big new multi-tenant client portal + ACG sales view. Did the unambiguous pieces, deferred the portal
to a feature request, then (on Howard's "b" greenlight) deployed to the live IX host and functionally
tested — which surfaced a production-breaking bug.

Bumped the wizard UI sizing ~10-12% (text, inputs, spacing, wider sheet, bigger buttons) and committed
it. Built `app/fieldcheck.cjs` and audited every questions.json field: 59 fields / 25 scorable, all 25
emit a usable finding on worst-case answers, no orphan/missing score keys, all findings complete, all 13
requiredControls resolve — every field yields usable info. Captured the portal vision as FR-1 in
`FEATURES.md` (personas, no-auto-sync guardrail, quote→active workflow, 3 auth options) and asked the
foundational auth decision; Howard chose to defer it to a feature request and focus on testing. Added an
executive summary + top-3 prioritized actions + ACG footer to the export (client = recommendations,
internal = + ACG service).

Deployed to IX (172.16.3.10, cPanel account `azcomputerguru`, docroot
`/home/azcomputerguru/public_html/security`, PHP 8.1.34) over SSH: backed up the live files, uploaded to
`.new`, lint-gated (php -l + JSON validate), then atomic-swapped into place. `config.php` (live secrets)
was never touched. The public URL was already live behind Cloudflare Access (a public hit returns CF
Access 403), so this was an update, not first-time setup.

Smoke-testing on the server found a **production-breaking bug**: `api.php`'s allow-list check did a
single `strcasecmp($email, ALLOWED_EMAIL)` against the WHOLE comma-separated list, so once a 2nd address
(howard@) was added on Jun 19, every API call (lookup/save/load/list/export) returned `forbidden` — the
live backend had been unusable. Fixed it to split + membership-check (matching index.php), redeployed,
and re-verified: `action=list` reads the DB; internal export renders posture/exec-summary/findings/ACG-
service/REQUIRED/captured-intake; client export renders posture+recommendations with zero upsell leaks
and no raw intake. Inserted a weak-answer test row, exported both views, then deleted it — prod DB back
to its 2 real rows. Committed everything to submodule `main` and advanced the claudetools pin so git
matches the live server.

## Key Decisions
- **Deploy via root SSH + base64 file upload, lint-gated atomic swap.** Backup → upload `.new` →
  `php -l`/JSON-validate → `mv` into place. Never overwrite `config.php`. Safe for a live site.
- **Functional-test via PHP CLI + a shipped harness, not inline `php -r`.** Inline `ssh "php -r '...'"`
  mangled through nested quoting (printed PHP usage); a base64'd `.php` harness that includes api.php and
  drives `$_GET[action]` is reliable. Bypasses the Apache/Cloudflare vhost routing (origin-direct curl
  404'd — a vhost/SNI artifact, not a broken deploy).
- **Test row written to the tool's own MySQL then deleted** — acceptable for a smoke test (its own DB,
  not Syncro/RMM), clearly tagged `ZZ-SMOKE-DELETE`, cleaned up.
- **Portal deferred to FR-1** (Howard's call) rather than building a multi-tenant auth model speculatively.
- **MSYS `ssh` for password+askpass**, not native Windows OpenSSH (which fails the .sh askpass) — matches
  the working pfsense-ssh.sh pattern.

## Problems Encountered
- **PROD BUG — api.php 403 for everyone** (single strcasecmp vs comma list). Found via live smoke test;
  fixed + redeployed + verified. This is why "test every function" mattered.
- **Native Windows OpenSSH askpass failed** `CreateProcessW error:193` (can't exec a .sh askpass) →
  used MSYS bare `ssh`. Logged --friction.
- **Inline `php -r` over ssh mangled** (nested quote stripping) → shipped a base64'd .php harness.
  Logged --friction (ref `feedback_windows_quote_stripping`).
- **Origin-direct curl 404'd** (vhost/SNI; CF proxies the real path) → tested via PHP CLI instead.
- **Submodule reset to old scaffold** earlier (stale gitlink + a concurrent submodule-update) → restored
  to origin/main and advanced the claudetools pin to stop the churn.

## Configuration Changes
security-assessment submodule (all on `main`, pushed; claudetools pin advanced each time):
- `app/index.php` — UI sizing bump (`66eb7cb`).
- `app/api.php` — export exec-summary/top-3/footer (`3e3a9ab`); **allow-list auth fix** (`f246091`).
- `FEATURES.md` (new) — FR-1 portal request; `app/fieldcheck.cjs` (new) — field-audit dev tool (`3a2301b`).
- Local scratch `app/_deploytest.php` + `app/_smoke.php` created for testing, removed after (not committed).
Live IX server: `index.php`, `api.php`, `questions.json` updated in the docroot (backups `.bak-20260621-181744`); `config.php` untouched.
ClaudeTools: this session log + `errorlog.md` (2 friction entries).

## Credentials & Secrets
None created/discovered. Used existing vault `infrastructure/ix-server` (root SSH 172.16.3.10:22, password
field `credentials.password`; also a WHM API token `credentials.whm-api-token` = full-access root). DB creds
live in the server's `config.php` (vault `msp-tools/security-assessment-db`). No secrets printed/committed.

## Infrastructure & Servers
- IX server 172.16.3.10 (ext 72.194.62.5), Rocky Linux WHM/cPanel, PHP 8.1.34 (ea-php81), root SSH :22.
- Site docroot: `/home/azcomputerguru/public_html/security` (cPanel acct `azcomputerguru`, sub of azcomputerguru.com, vhost 172.16.3.10:80/:443).
- security.azcomputerguru.com behind Cloudflare Access (Zero Trust app `8ce5f31c-...`; allow mike@ + howard@). Origin answers via CF; origin-direct curl needs `--resolve`/Host and still 404'd (CF-proxied path is the working one).
- Live DB (acgsec_assess): 2 real rows (id 1 Darrell Delphen, id 2 empty) after test cleanup.

## Commands & Outputs
```
# find docroot
ssh root@172.16.3.10 'grep -i security.azcomputerguru.com /etc/userdatadomains'
#   -> azcomputerguru==root==sub==azcomputerguru.com==/home/azcomputerguru/public_html/security==...ea-php81
# deploy pattern: backup -> base64 upload to .new -> php -l -> chown user -> mv (atomic)
base64 local.php | ssh root@ix "base64 -d > docroot/file.new"
# functional test (bypasses Apache/CF): shipped harness includes api.php
php _deploytest.php list                 # {"items":[{id:2...},{id:1,...}]}  (after the auth fix)
php _deploytest.php export <id> internal # posture/exec/findings/ACG service/Captured intake
php _deploytest.php export <id> client   # posture/recommendations, 0 ACG-service, no Captured intake
# BUG before fix: php _deploytest.php list -> {"error":"forbidden"}
```

## Pending / Incomplete Tasks
- **Live browser UI click-through** (through Cloudflare Access) — the only thing not tested; needs a human
  logged into https://security.azcomputerguru.com (look up a client, answer fields, watch posture, Export
  both views). Backend fully verified server-side.
- **#1 GuruRMM endpoint prefill** — deferred (infra: no Syncro→RMM mapping, no reachable RMM API from IX).
- **FR-1 multi-tenant portal** — filed in FEATURES.md, awaiting the auth decision when Howard wants it.

## Reference Information
- Submodule `main` HEAD `f246091` (claudetools pin `27c1d97` at time of fix).
- Commits this turn: `66eb7cb` sizing, `3a2301b` FR-1+fieldcheck, `3e3a9ab` export exec-summary, `f246091` api auth fix.
- Live server backups: `*.bak-20260621-181744` in the docroot (rollback if needed).
- Export endpoint: `api.php?action=export&id=<id>&view=internal|client` (origin requires the CF Access email header).
- Companion logs: `2026-06-21-howard-security-assessment-scoring.md`, `...-unifi-pfsense-control-verbs.md`, `...-gururmm-bug-018-019.md`.

---

## Update: 18:53 PT — site was broken in browser (stale opcache); root-caused + fixed; CONFIRMED loading

After the deploy, Howard reported the live page "mostly broken — only the Export and Assessments
buttons at the top." Diagnosed and resolved.

### What was wrong (two compounding issues)
1. **api.php allow-list bug** (already fixed earlier this session): single `strcasecmp($email, ALLOWED_EMAIL)`
   vs the whole comma string → 403 for every API call. Confirmed live via the access log (`POST /api.php?action=save 403` repeatedly).
2. **Stale PHP OPcache** — the server kept executing the OLD `index.php` (rendered output 14637 bytes) and OLD
   `api.php` even though the new files were on disk (md5 matched local). So the new wizard never rendered (boot()
   ran the old code / the old api 403'd) and the fix wasn't live.

### Diagnosis path
- Verified deployed files = local (identical md5), perms 644, `.htaccess` benign, questions.json valid on server,
  no PHP error log → not a crash. Ran the wizard JS in node with DOM/fetch stubs → boot() completed fine (so not
  a code bug). The **Apache SSL access log** (`/etc/apache2/logs/domlogs/security.azcomputerguru.com-ssl_log`) was
  the smoking gun: `GET / 200 14637` (old size; new index.php renders ~27.9k) + many `api.php ... 403`.
- SAPI = Apache (httpd) + opcache On. Origin-direct curl to 127.0.0.1 hits a default vhost that 404s — the app is
  only reachable via the EXTERNAL IP path Cloudflare uses.

### Fix
- Dropped a one-off `_oc.php` (`opcache_reset()`) in the docroot and ran it through the real path:
  `curl -sk --resolve security.azcomputerguru.com:443:72.194.62.5 -H 'Cf-Access-Authenticated-User-Email: mike@azcomputerguru.com' https://.../_oc.php` → `opcache_reset: OK` (validate_timestamps=1, but the cache hadn't revalidated; reset forced it).
- Verified: `GET /` now 200 **27885 bytes**, served HTML contains `computeScores`/`gradeFor`/"Posture &amp; findings";
  `api.php?action=list` returns JSON (no 403). Removed `_oc.php` + local scratch (`_jsrun.cjs`, `_deploytest.php`, `_smoke.php`).
- Howard confirmed: **"it is loading now."**

### Durable fixes
- `DEPLOY.md` updated (`31bc786`): "FLUSH OPCACHE after updating" (the external-IP `_oc.php` recipe) + the
  comma-separated `ALLOWED_EMAIL` gotcha. claudetools pin advanced (`5299f3d`). Logged --friction (deploy/cpanel: stale opcache).

### Key gotchas for next deploy
- After uploading PHP changes to this cPanel host, opcache can serve stale bytecode → flush it (recipe in DEPLOY.md).
- Origin-direct (127.0.0.1) 404s the vhost; test/flush via `--resolve ...:443:72.194.62.5` (external IP).
- Coord (Mike, GURU-5070): BUG-018 (cea87d4) + Event Log Watch UI merged to gururmm main + deploying; I'm cleared to self-merge gururmm going forward.

### Net state
security.azcomputerguru.com is LIVE and working (new scoring wizard + larger UI + dual export + fixed backend),
verified server-side and confirmed loading in Howard's browser. Remaining: #1 RMM prefill (deferred, infra), FR-1 portal (deferred, auth decision).

---

## Update: 19:31 PT — post-deploy live fixes (wrong-customer lookup bug + New/clear button)

Two issues Howard found while using the live site, both fixed + deployed + verified.

### Bug: phone lookup returned the WRONG customer
- Symptom: entering `5205851310` pulled up "Marjolaine Deslauriers" (an unrelated record).
- Root cause: Syncro's `GET /customers?phone=<n>` param does NOT filter — it returns the full customer
  list (5079 across 51 pages); the code took `customers[0]` (Marjolaine, phone 5203310273). Reproduced
  via the Syncro API (vault `msp-tools/syncro-howard` cred field `credentials.credential`): `?phone=` →
  count 100 unfiltered; `?query=5205851310` → count 0 (number not in Syncro); `?query=5208880000`
  (Insty-Prints' real number) → count 1 (so `?query=` DOES search phone by digits).
- Fix (api.php lookup): use `?query=<digits>` and VERIFY a candidate's phone digit-tail matches the
  entered digits before prefilling (handles leading-1 / 7-vs-10 digits). No match → return a clear
  "No customer matches that phone number" + suggestions, and prefill NOTHING. index.php doLookup updated
  to show the message/suggestions and not prefill on no-match. Format was never the issue (input is
  digit-normalized, so 5205851310 == (520) 585-1310).
- Verified live (web POST via the external-IP path): `5205851310` → no-match; `5208880000` → Insty-Prints
  full prefill; `(520) 585-1310` → no-match (same as digits).

### Feature: "+ New" button (Howard's request)
- After loading a client, `state.id` persisted, so a fresh lookup + save could overwrite the previous
  client's record. Added a top-bar **+ New** button -> `clearAll()` resets id/data/consent/sec/view +
  URL + topbar, so the next save INSERTs a new record. Confirms before clearing if work is in progress.

### Deploy mechanics (each fix)
- Upload changed file(s) to `.new` -> `php -l` -> atomic `mv` -> flush opcache via a one-off `_oc.php`
  hit through `--resolve ...:443:72.194.62.5` (external IP; 127.0.0.1 vhost 404s) -> verify served HTML.
  Backups `*.bak-<ts>` in the docroot. Browser needs Ctrl+Shift+R to pick up index.php changes.

### Commits
- `ef03b4e` lookup phone-match fix; `a9c85a7` + New button. claudetools pins advanced (`c462f50`, `5b44a3e`).

### Net
Phone search works correctly (verified by Howard), wrong-customer risk eliminated, and the + New button
prevents cross-record contamination. Site fully functional.

---

## Update: 21:25 PT — onboarding/upsell questions + opportunity engine + Save-quote + quote delete

Expanded the assessment per Howard ("what other questions to better integrate clients + gather upsell
data") plus quote lifecycle controls. All deployed to IX + verified live.

### Save-quote + status
- Added a top-bar **Save quote** button + QUOTE badge; added a `status` column (default `quote`) to the
  `assessments` table (ALTER on live DB + schema.sql). New saves are `quote` (unverified, never auto-synced);
  UPDATE leaves status (preserves a future promotion). `list` returns status; Assessments list shows it.

### New questions (Howard's "go with recommendations")
- Security-scored: **M365 license tier** (Basic/Standard -> upgrade to Business Premium; it unlocks MFA/CA/Intune/Defender).
- New **Other Services** section: VoIP (PacketDial), managed print, surveillance, web hosting.
- New **Onboarding & Sales Context** section (internal-only): current support model, existing agents to remove,
  admin-access readiness, hardware age, cyber-insurance renewal timing, decision-maker, biggest frustration, planned changes.
- Plus WiFi coverage, internet failover, UPS.

### Opportunity engine (the key design)
- New field-level `opportunity` block (parallel to `risk`): non-security upsells that fire on a `when`
  trigger ({equals|in|includes|notIncludes}) and surface for SALES WITHOUT affecting the security grade.
- Engine mirrored JS (`computeScores` returns `opportunities`) + PHP (`score_assessment` + `opp_fires`).
- Export: INTERNAL shows an "Upsell opportunities" section (each -> ACG service); CLIENT view hides them
  entirely. Wizard internal view shows them; client toggle hides. Verified: 6 opportunities fire on a weak
  sample, security findings stay separate (2).

### Quote lifecycle
- api `promote` (quote->active) + `delete` actions (ACG-gated). Assessments list: **Delete** works
  (confirm + clears the loaded record if it's the one deleted); **Activate/make-live button DISABLED** per
  Howard (live import to Syncro/RMM stays manual until wired; promote API is dormant/ready).

### Problem hit
- First deploy shipped api.php + index.php but NOT the new questions.json -> opportunities/new questions didn't
  appear live (OPPS=0). Root-caused via a CLI opp-debug (server questions.json was old). Deployed questions.json
  (lint-gated) -> verified opportunities fire + new sections live. Lesson: deploy ALL changed files, incl. data.

### Commits / state
- Submodule `0f6927b` (questions+engine+delete) on main; earlier this session `66eb7cb`/`3a2301b`/`3e3a9ab`/
  `f246091`/`a9c85a7`/`7a3dfb8`. claudetools pin `0f6927b` (HEAD `8713b46`). All live on IX, opcache flushed.
- Concurrent screenconnect WIP in the shared tree preserved through the pin rebases.

### Still open
- Activate/promote UI intentionally disabled (wire live import later). FR-1 portal still deferred (auth decision).
- #1 RMM prefill deferred (infra). Optional: extend fieldcheck.cjs to validate `opportunity` fields.