# websvr→IX hosting cleanup, DNS-safe account removals, furrier reseller, Ralph's Live Transfer, amtransit/Neptune

## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin

## Session Summary

Large multi-front session dominated by ACG legacy-hosting cleanup and migration on **websvr.acghosting.com** (CloudLinux 7.9, cPanel 110) toward **IX** (ix.azcomputerguru.com, CloudLinux 9, cPanel 136). websvr was at 96% disk (42 GB free); root cause is Ralph's Transfer's 542 GB mailbox account. First trimmed the ext4 root reserve from 5%→1% (`tune2fs -m 1 /dev/xvda2`, +~40 GB usable), then ran a DNS-safe cleanup: scanned all 239 domains across both servers, resolved public A/MX for 194, classified each by account-server vs DNS location, and cross-checked with per-domain Apache traffic logs to confirm which sites each server actually serves.

Removed **18 stale duplicate accounts** from websvr (accounts whose live copy is on IX/M365/external, zero web traffic on websvr, no live mail there). All removals used `whmapi1 removeacct user=<u> keepdns=1` — critical because IX+websvr share a **cPanel DNS cluster in "sync" mode**, so a normal removeacct's zone deletion would propagate cluster-wide and take domains dark. `keepdns=1` skips the DNS kill entirely (validated on a controlled test with atlantisbevcollc.com: account removed, zones on IX/AWS/public untouched). All 20 affected zones were backed up (websvr + IX copies) to the repo first.

For sole-copy accounts, established an **archive-then-remove** pattern: `pkgacct` (files+mail+DBs) → **Jupiter** (Unraid, 172.16.3.20) pulls the tarball into `/mnt/user/Backups/websvr-archives/` via temporary SSH key → verify md5 byte-identical on both ends + gzip integrity → `removeacct keepdns=1`. Applied to **compoundfitness** (dead domain, 1.48 GB archive), **grabbanddurando** (12 GB gdapp/data.grabblaw.com stale copy + 29 DBs, 8.9 GB archive), **tucsonsafety** (decommissioned client, 3.0 GB) and **devgrabblaw** (dev site, 9.8 GB). websvr went **96%→78%** (216 GB free).

Created the **`furrier` reseller on IX** (`furrier.acghosting.com`, standard self-manage ACLs, password `Palo3705S`, vaulted) to fix 7 Furrier-family accounts (westernt, jackfurr, jefffurr, sparcora, thecentu, ultraper, upracing) that were orphan-owned by a non-existent `furrier` owner — now properly attached (8 accounts). Planned Ralph's migration; Mike kicked off a **cPanel Live Transfer** (IX pulls ralphstransfer from websvr, session `websvracghosticopya20260717232750wf4`) which is running (~434 Mbps ≈ websvr's 480 Mbps cap, ETA ~7:15–7:30 PM). Finally, located **amtransit.com** mail (not on IX/websvr) on **Neptune Exchange** — confirmed via a GuruRMM on-box AD `proxyAddresses` lookup (Neptune WinRM needs VPN), returning 6 mailboxes, and updated the amtransit wiki.

Earlier in the day (separate committed log `clients/azcomputerguru.com/session-logs/2026-07-16-mike-mailbox-security-investigation-exchange-tier-breakglass.md`): mike@ mailbox security investigation, the Exchange-tier (`exchange-op` vs `investigator-exo`) fix, WHfB pruning, and break-glass CA exclusions. Also this session: Khalsa Montessori RMM onboarding (17 SC→RMM), BirthBiologic SharePoint permissions audit → Syncro #32187 comment, Apple supervision cert vaulted, Reliant Google-Business-Profile finding (none exists), and the desertrat/Furrier login incident (server healthy; recovered).

## Key Decisions
- **`removeacct keepdns=1` instead of editing DNS cluster roles.** Cleaner than temporarily setting websvr's cluster role to standalone — same outcome (no zone-delete propagation), zero risk to the live cluster config that serves all domains. Validated on one account before batching.
- **Archive to Jupiter via Jupiter-pull, not a 500 GB tarball or through the workstation.** pkgacct + Jupiter rsync-pull (temp key) keeps data server-to-server; md5-verified both ends before any removal.
- **Backed up all zones (both servers) before removals** even though keepdns=1 preserves them — safety net, committed to repo.
- **tucsonsafety removal gated on explicit confirmation** — it's a live client with external DNS (Network Solutions) we can't redirect; only removed after Mike confirmed decommission.
- **furrier reseller main domain = furrier.acghosting.com** (controlled placeholder), standard self-manage ACLs — upr.com becomes its real domain when it migrates next month.
- **Ralph's via cPanel Live Transfer (Mike's call)** — native sync-during-copy beats manual config-first+rsync; DNS cutover clean since we control ralphstransfer.com (our cluster).
- **amtransit via RMM on-box AD lookup** (Mike's suggestion) — bypassed the Neptune VPN/WinRM barrier; SYSTEM/computer-account can read AD proxyAddresses without Exchange RBAC.

## Problems Encountered
- **`view_transfer` path mangled by Git-Bash MSYS** (`C:/Program: No such file`) — `/usr/local/...` args auto-converted. Fixed by base64-encoding remote commands (`echo <b64> | base64 -d | bash`).
- **pkgacct "completion" false-positives** — my wait loops matched the wrong process-name pattern (actual proc is `pkgacct - <user> - av: 4 - ...` with dashes) and sampled mid-write tarballs (bogus 149 MB / gzip-fail). Fixed pattern; verified via stable size + gzip -t.
- **Ralph's transfer "PID gone" false alarm** — cPanel transfers re-parent between copy and restore phases, so the initial master PID exits. Replaced the PID-keyed monitor with a session-state monitor (`view_transfer` "is running" header).
- **Long-held SSH monitor session dropped** ("Network error: Software caused connection abort"). Rebuilt monitors to use short per-cycle connections instead of one long-held session.
- **Neptune unreachable via WinRM from GURU-5070** (port 5985 closed; needs VPN). Worked around via GuruRMM agent on-box.
- **grabbanddurando initially mis-diagnosed** as "broken on IX / incomplete migration" — I checked a stale docroot (`/public_html/data`, empty). The real app is at `/public_html/data_grabbanddurando/gdapp` (16 GB) with all DBs; data.grabblaw.com serves 200 on IX. Corrected.
- **Vault redact filter leaked a SA private key** to transcript (grep denylist missed `private_key`). Logged as friction; noted allowlist approach.

## Configuration Changes
- websvr `/dev/xvda2`: ext4 reserved-blocks 5%→1% (`tune2fs -m 1`), +~40 GB usable.
- websvr accounts removed (all keepdns=1): atlantisbevco, gurushow, danaise, farwest, jrkco, bruceext, peacefulspirit, grabblaw, arizonahatters, desertfox, fsgtucson, nwpool, cryoweave, antivaxxer, westernt, azcomputerguru, azrestaurant, outaboundssports, compoundfitness, grabbanddurando, tucsonsafety, devgrabblaw (22 total accounts removed across the session; 18 stale-duplicate + 4 archived).
- IX: created `furrier` reseller (createacct + setupreseller makeowner=1 + setacls standard).
- Repo: `.claude/skills/remediation-tool/SKILL.md`, `references/gotchas.md`, `scripts/user-breach-check.sh` (Exchange-tier fix); `scripts/breakglass-signin-alert.sh` + `run-breakglass-alert.cmd` (new); `.claude/memory/reference_exchange_online_exchangeop_tier.md` (new) + MEMORY.md; `clients/internal-infrastructure/dns-zone-backups/2026-07-17-websvr-safe-cleanup/` (40 zone files + manifest); `wiki/clients/arizona-medical-transit.md` (Neptune + mailboxes); root `.gitignore` (scratch filters) + swept ~5.3 MB tracked root scratch.
- GURU-5070: scheduled task `ACG-BreakGlass-SignIn-Alert` (every 4h).

## Credentials & Secrets
- **furrier reseller (IX):** user `furrier`, password `Palo3705S`, WHM https://ix.azcomputerguru.com:2087 — vaulted `clients/furrier/ix-cpanel-reseller.sops.yaml`.
- **Apple supervision cert:** passphrase `caCNnQBVbq` + base64 of SupervisionIdentityCertificate.p12 (CN=mdm_supervision_identity) — vaulted `msp-tools/apple-device-pairing-cert.sops.yaml`.
- **Khalsa RMM site key:** site_code `LOWER-TIGER-5022` — vaulted `clients/khalsa/gururmm-site-main.sops.yaml`.
- websvr root, IX (key `~/.ssh/id_ed25519`), Jupiter (key), Neptune (`ACG\administrator`, WinRM/VPN) — existing vault entries.

## Infrastructure & Servers
- **websvr.acghosting.com** — 162.248.93.81/.233/.78, CloudLinux 7.9, cPanel 110, 1 TB xvda2, 480 Mbps cap. Now 78% (216 GB free).
- **IX** — ix.azcomputerguru.com, int 172.16.3.10 / ext 72.194.62.5, cPanel 136, /home 4.4 TB (2.7 TB free).
- **Jupiter** — 172.16.3.20, Unraid, backups share `/mnt/user/Backups/websvr-archives/` (97 TB, 31 TB free).
- **Neptune Exchange** — neptune.acghosting.com, int 172.16.3.11 / ext 67.206.163.124, Exchange 2016, at Dataforth D2; WinRM needs VPN; RMM agent `b3a9b454-86eb-491c-ac67-c1f98987d8dc` (online). Hosts amtransit.com (6 mailboxes).
- **DNS cluster** (all role=`sync`): websvr, IX (72.194.62.5), ns2.acghosting.com=52.52.94.202 (AWS), 184.187.220.69. Our cluster NS: ns1.acghosting.com / ns2.acghosting.com / ns1/ns2.azcomputerguru.com.
- **Ralph's Live Transfer session:** `websvracghosticopya20260717232750wf4` (IX←websvr, running).

## Commands & Outputs
- Zone dump/restore-ready: `/var/named/<domain>.db` per account; backups in repo + `ix-copies/`.
- Cluster mode: `cat /var/cpanel/cluster/root/config/*-dnsrole` → `sync`.
- keepdns removal: `whmapi1 removeacct user=<u> keepdns=1`.
- Archive: `/scripts/pkgacct <u>` → `/home/cpmove-<u>.tar.gz`; Jupiter pull: `rsync -a -e "ssh -i /root/.ssh/id_ed25519" root@162.248.93.81:/home/cpmove-<u>.tar.gz /mnt/user/Backups/websvr-archives/`.
- Restore any archive: `/scripts/restorepkg cpmove-<u>` (drop tarball in /home first).
- Transfer status: `/usr/local/cpanel/bin/view_transfer websvracghosticopya20260717232750wf4`.
- amtransit AD lookup (RMM cmd `94b9a1a6-…` on NEPTUNE): `([adsisearcher]"(proxyAddresses=*@amtransit.com*)").FindAll()` → 6 mailboxes: driver@, driver-ext@, bertie@ (Ana Lozano), matilde@ (Matilde Slate), nydia@, caitlin@.
- Bandwidth (transfer live): websvr eth0 TX ~434 Mbps; IX eno3 RX ~444 Mbps.

## Pending / Incomplete Tasks
- **Ralph's Live Transfer in progress** (session `…232750wf4`) — background monitor `bdam0du3c` running (session-state based; pings on completion/fail/stall/websvr-low). On completion: DNS cutover — repoint ralphstransfer.com A/MX to IX (our cluster). ralphsmoving.com web is on client Cloudflare (separate).
- **Furrier moves parked to next month:** migrate upr.com (`furrier`) + desertrat (`desertra`) websvr→IX under the furrier reseller; fix upr.com's stray owner tag during the move.
- **amtransit [BILLING]:** 6 mailboxes exist vs 5 billed "Exchange Hosted Email" — reconcile (Winter).
- **BirthBiologic SharePoint remediation** (Syncro #32187) awaiting Annise's coordinator list; tenant default-link change + Donor Services cleanup staged, not executed.
- **mike@ break-glass:** rotate sysadmin@azcomputerguru.onmicrosoft.com password in Entra portal (Graph 403 by design), then vault at `msp-tools/acg-m365-breakglass.sops.yaml`; also exclude from admin-MFA policy already done.
- **Reliant Google Business Profile:** none exists — needs creation/verification before a reviews link/QR is possible.
- **Wiki:** run `/wiki-compile client:internal-infrastructure --full` and `/wiki-compile client:arizona-medical-transit --full` to fold in this session.

## Reference Information
- Ralph's transfer session id: `websvracghosticopya20260717232750wf4`
- NEPTUNE RMM agent: `b3a9b454-86eb-491c-ac67-c1f98987d8dc`; amtransit query cmd `94b9a1a6-6148-4331-afe4-7423a20486dd`
- Khalsa RMM: client `ae08187d-bd2b-43d7-83ac-d82e4fadd6b2`, site `11ef9472-1405-4767-b095-7cf2dff0bc3c`, code LOWER-TIGER-5022
- BB SharePoint: Syncro #32187 (id 109277420), comment id 424114287; Donor Services site /sites/donorservices, group `11d28c77-c259-4429-8aab-a3a3945f58fd`
- Jupiter archives: /mnt/user/Backups/websvr-archives/ — compoundfitness (befef13f…), grabbanddurando (f9a06482…), tucsonsafety (b78b0799…), devgrabblaw (92641a2f…)
- Related log: clients/azcomputerguru.com/session-logs/2026-07-16-mike-mailbox-security-investigation-exchange-tier-breakglass.md

---

## Update: 2026-07-18 06:10 PT — Ralph's transfer verified + cutover confirmed COMPLETE (mike)

Continuation of the Ralph's Live Transfer started 7/17. Verified the transfer landed cleanly, established the DNS cutover had already happened automatically, confirmed zero mail loss, and posted a team bot-alert. Ralph's migration is DONE; only the websvr-side decommission remains (deferred as a rollback cushion).

### What was done
1. **Transfer completion verified.** `view_transfer` = "Account ralphstransfer: Success / Session Complete" (7/17 19:26). IX account live, not suspended, 542 GB mail, all 25 mailboxes present (applications, armando, billing, claims, dispatch, info, kathleen.wicks, kathryn, kevin.razo, lw, lynette, managers, nts, office, olivia, phones, pvo15803, rob.wicks, sales, survey, taylor.chaidez, tc, test, warehouse, whse).
2. **DB-count flag resolved — false alarm.** Earlier "6 DBs on source vs 2 on IX" was a mis-read. Both servers have exactly **2 DBs** (`ralphstr_newwp`, `ralphstr_s597`) and **2 DB users** (`ralphstr_admin`, `ralphstr_s597`) — identical. Read `/var/cpanel/databases/ralphstransfer.json` on both to confirm. No gap.
3. **ralphsmoving.com "missing addon" flag resolved — vestigial.** ralphsmoving.com is in websvr `/etc/userdomains` + `DNS1=` in the user file, but **NOT** in `parked_domains`/`addon_domains` (userdata/main both empty) → no vhost, no local named zone, no docroot, no mail (all 25 mailboxes are on ralphstransfer.com). It did not (correctly) transfer to IX. The **actual** ralphsmoving.com behavior is a **Cloudflare 301 redirect** → `https://ralphstransfer.com/` (NS crystal/bradley.ns.cloudflare.com; A 172.67.216.45 / 104.21.45.152; `Server: cloudflare` on the 301). Because the CF redirect targets the *hostname* ralphstransfer.com (not an IP), it kept working through the cutover with zero changes.
4. **Cutover confirmed ALREADY DONE — rode along with the restore.** `ralphstransfer.com` A = **72.194.62.5 (IX)**, MX = ralphstransfer.com (itself → IX), NS ns1/ns2.azcomputerguru.com. Both IX and websvr zone files were rewritten at the **exact same instant `2026-07-17 19:24:46`** with IX's IP — the Live Transfer restore wrote the zone pointing at IX and the DNS cluster (sync mode) propagated it. IX zone serial 2026071804.
5. **Zero mail lost — no delta needed.** websvr received **0** inbound since 19:00 7/17; its **newest message is 16:26 7/17** (before the transfer's mail copy AND before the flip), so everything it ever received was captured. IX has been the live server continuously — 95 deliveries in the current log, newest mail file 04:43 7/18 (rob.wicks, kathleen.wicks/docusign, lynette, etc.). The pre-compaction "final delta sync" concern is MOOT — the flip rode along with the restore, so there was never a split-delivery window.
6. **Source already suspended by the Transfer Tool.** `/var/cpanel/suspended/ralphstransfer` created 19:24:46, reason "User transferred to another server"; `whmapi accountsummary` = `suspended: 1`. So at 19:24:46 three things happened together: DNS→IX, source suspended, transfer complete. No manual suspend needed.
7. **Decommission deferred per Mike** — leave the suspended websvr copy as a rollback cushion for a few days (~7/21), then `removeacct user=ralphstransfer keepdns=1`.
8. **Bot-alert posted** to #bot-alerts (message_id 1528012554029502634) so Rob/team have the migration status.

### Key facts / decisions (this update)
- The cPanel Live Transfer "Copy an Account" auto-suspends the source AND rewrites the DNS zone to the destination IP on restore — in a shared DNS cluster (sync mode) that IS the cutover. No separate manual DNS flip was required (contrary to the pre-compaction plan).
- Decommission MUST use `keepdns=1` — a plain `removeacct` on websvr would propagate a zone deletion cluster-wide and kill ralphstransfer.com on IX. keepdns=1 leaves IX's authoritative zone intact.
- ralphsmoving.com needs nothing: Cloudflare owns its DNS + the redirect, which follows the ralphstransfer.com hostname.

### Commands / outputs (this update)
- IX check: `ssh -i ~/.ssh/id_ed25519 ix "echo <b64> | base64 -d | bash"` — DB map, `/etc/userdomains`, userdata/main (parked/addon), `/var/named/ralphstransfer.com.db` mtime+serial, exim_mainlog greps.
- websvr check: `plink -hostkey SHA256:qcaW8BWq5UyM0l0g6DS9JfYbMZN/LTXLs3BIEZV8BE0 -pw <vault infrastructure/websvr-legacy-hosting> -batch root@162.248.93.81 "echo <b64> | base64 -d | bash"` — same fields + suspend flag.
- Live DNS/HTTP: `nslookup -type=A ralphstransfer.com 8.8.8.8` → 72.194.62.5; `curl -sI https://ralphsmoving.com` → 301 Location: https://ralphstransfer.com/ Server: cloudflare.
- Bot-alert: `bash .claude/scripts/post-bot-alert.sh "<msg>" bot` → message_id 1528012554029502634.

### Pending (updated)
- **Ralph's websvr decommission (~2026-07-21):** `whmapi1 removeacct user=ralphstransfer keepdns=1` on websvr → reclaims 542 GB (websvr ~78% → ~35%). Suspended copy is the rollback cushion until then.
- (Unchanged from 7/17) Furrier upr.com + desertrat moves parked to next month; amtransit 6-vs-5 mailbox billing reconcile (Winter); BB SharePoint #32187 awaiting Annise's list; mike@ break-glass password rotation; Reliant GBP creation; wiki compiles.
