claudetools/clients/internal-infrastructure/session-logs/2026-07/2026-07-17-mike-websvr-ix-hosting-cleanup-migration.md

websvr→IX hosting cleanup, DNS-safe account removals, furrier reseller, Ralph's Live Transfer, amtransit/Neptune

User

• User: Mike Swanson (mike)
• Machine: GURU-5070
• Role: admin

Session Summary

Large multi-front session dominated by ACG legacy-hosting cleanup and migration on websvr.acghosting.com (CloudLinux 7.9, cPanel 110) toward IX (ix.azcomputerguru.com, CloudLinux 9, cPanel 136). websvr was at 96% disk (42 GB free); root cause is Ralph's Transfer's 542 GB mailbox account. First trimmed the ext4 root reserve from 5%→1% (tune2fs -m 1 /dev/xvda2, +~40 GB usable), then ran a DNS-safe cleanup: scanned all 239 domains across both servers, resolved public A/MX for 194, classified each by account-server vs DNS location, and cross-checked with per-domain Apache traffic logs to confirm which sites each server actually serves.

Removed 18 stale duplicate accounts from websvr (accounts whose live copy is on IX/M365/external, zero web traffic on websvr, no live mail there). All removals used whmapi1 removeacct user=<u> keepdns=1 — critical because IX+websvr share a cPanel DNS cluster in "sync" mode, so a normal removeacct's zone deletion would propagate cluster-wide and take domains dark. keepdns=1 skips the DNS kill entirely (validated on a controlled test with atlantisbevcollc.com: account removed, zones on IX/AWS/public untouched). All 20 affected zones were backed up (websvr + IX copies) to the repo first.

For sole-copy accounts, established an archive-then-remove pattern: pkgacct (files+mail+DBs) → Jupiter (Unraid, 172.16.3.20) pulls the tarball into /mnt/user/Backups/websvr-archives/ via temporary SSH key → verify md5 byte-identical on both ends + gzip integrity → removeacct keepdns=1. Applied to compoundfitness (dead domain, 1.48 GB archive), grabbanddurando (12 GB gdapp/data.grabblaw.com stale copy + 29 DBs, 8.9 GB archive), tucsonsafety (decommissioned client, 3.0 GB) and devgrabblaw (dev site, 9.8 GB). websvr went 96%→78% (216 GB free).

Created the furrier reseller on IX (furrier.acghosting.com, standard self-manage ACLs, password Palo3705S, vaulted) to fix 7 Furrier-family accounts (westernt, jackfurr, jefffurr, sparcora, thecentu, ultraper, upracing) that were orphan-owned by a non-existent furrier owner — now properly attached (8 accounts). Planned Ralph's migration; Mike kicked off a cPanel Live Transfer (IX pulls ralphstransfer from websvr, session websvracghosticopya20260717232750wf4) which is running (~434 Mbps ≈ websvr's 480 Mbps cap, ETA ~7:15–7:30 PM). Finally, located amtransit.com mail (not on IX/websvr) on Neptune Exchange — confirmed via a GuruRMM on-box AD proxyAddresses lookup (Neptune WinRM needs VPN), returning 6 mailboxes, and updated the amtransit wiki.

Earlier in the day (separate committed log clients/azcomputerguru.com/session-logs/2026-07-16-mike-mailbox-security-investigation-exchange-tier-breakglass.md): mike@ mailbox security investigation, the Exchange-tier (exchange-op vs investigator-exo) fix, WHfB pruning, and break-glass CA exclusions. Also this session: Khalsa Montessori RMM onboarding (17 SC→RMM), BirthBiologic SharePoint permissions audit → Syncro #32187 comment, Apple supervision cert vaulted, Reliant Google-Business-Profile finding (none exists), and the desertrat/Furrier login incident (server healthy; recovered).

Key Decisions

• removeacct keepdns=1 instead of editing DNS cluster roles. Cleaner than temporarily setting websvr's cluster role to standalone — same outcome (no zone-delete propagation), zero risk to the live cluster config that serves all domains. Validated on one account before batching.
• Archive to Jupiter via Jupiter-pull, not a 500 GB tarball or through the workstation. pkgacct + Jupiter rsync-pull (temp key) keeps data server-to-server; md5-verified both ends before any removal.
• Backed up all zones (both servers) before removals even though keepdns=1 preserves them — safety net, committed to repo.
• tucsonsafety removal gated on explicit confirmation — it's a live client with external DNS (Network Solutions) we can't redirect; only removed after Mike confirmed decommission.
• furrier reseller main domain = furrier.acghosting.com (controlled placeholder), standard self-manage ACLs — upr.com becomes its real domain when it migrates next month.
• Ralph's via cPanel Live Transfer (Mike's call) — native sync-during-copy beats manual config-first+rsync; DNS cutover clean since we control ralphstransfer.com (our cluster).
• amtransit via RMM on-box AD lookup (Mike's suggestion) — bypassed the Neptune VPN/WinRM barrier; SYSTEM/computer-account can read AD proxyAddresses without Exchange RBAC.

Problems Encountered

• view_transfer path mangled by Git-Bash MSYS (C:/Program: No such file) — /usr/local/... args auto-converted. Fixed by base64-encoding remote commands (echo <b64> | base64 -d | bash).
• pkgacct "completion" false-positives — my wait loops matched the wrong process-name pattern (actual proc is pkgacct - <user> - av: 4 - ... with dashes) and sampled mid-write tarballs (bogus 149 MB / gzip-fail). Fixed pattern; verified via stable size + gzip -t.
• Ralph's transfer "PID gone" false alarm — cPanel transfers re-parent between copy and restore phases, so the initial master PID exits. Replaced the PID-keyed monitor with a session-state monitor (view_transfer "is running" header).
• Long-held SSH monitor session dropped ("Network error: Software caused connection abort"). Rebuilt monitors to use short per-cycle connections instead of one long-held session.
• Neptune unreachable via WinRM from GURU-5070 (port 5985 closed; needs VPN). Worked around via GuruRMM agent on-box.
• grabbanddurando initially mis-diagnosed as "broken on IX / incomplete migration" — I checked a stale docroot (/public_html/data, empty). The real app is at /public_html/data_grabbanddurando/gdapp (16 GB) with all DBs; data.grabblaw.com serves 200 on IX. Corrected.
• Vault redact filter leaked a SA private key to transcript (grep denylist missed private_key). Logged as friction; noted allowlist approach.

Configuration Changes

• websvr /dev/xvda2: ext4 reserved-blocks 5%→1% (tune2fs -m 1), +~40 GB usable.
• websvr accounts removed (all keepdns=1): atlantisbevco, gurushow, danaise, farwest, jrkco, bruceext, peacefulspirit, grabblaw, arizonahatters, desertfox, fsgtucson, nwpool, cryoweave, antivaxxer, westernt, azcomputerguru, azrestaurant, outaboundssports, compoundfitness, grabbanddurando, tucsonsafety, devgrabblaw (22 total accounts removed across the session; 18 stale-duplicate + 4 archived).
• IX: created furrier reseller (createacct + setupreseller makeowner=1 + setacls standard).
• Repo: .claude/skills/remediation-tool/SKILL.md, references/gotchas.md, scripts/user-breach-check.sh (Exchange-tier fix); scripts/breakglass-signin-alert.sh + run-breakglass-alert.cmd (new); .claude/memory/reference_exchange_online_exchangeop_tier.md (new) + MEMORY.md; clients/internal-infrastructure/dns-zone-backups/2026-07-17-websvr-safe-cleanup/ (40 zone files + manifest); wiki/clients/arizona-medical-transit.md (Neptune + mailboxes); root .gitignore (scratch filters) + swept ~5.3 MB tracked root scratch.
• GURU-5070: scheduled task ACG-BreakGlass-SignIn-Alert (every 4h).

Credentials & Secrets

• furrier reseller (IX): user furrier, password Palo3705S, WHM https://ix.azcomputerguru.com:2087 — vaulted clients/furrier/ix-cpanel-reseller.sops.yaml.
• Apple supervision cert: passphrase caCNnQBVbq + base64 of SupervisionIdentityCertificate.p12 (CN=mdm_supervision_identity) — vaulted msp-tools/apple-device-pairing-cert.sops.yaml.
• Khalsa RMM site key: site_code LOWER-TIGER-5022 — vaulted clients/khalsa/gururmm-site-main.sops.yaml.
• websvr root, IX (key ~/.ssh/id_ed25519), Jupiter (key), Neptune (ACG\administrator, WinRM/VPN) — existing vault entries.

Infrastructure & Servers

• websvr.acghosting.com — 162.248.93.81/.233/.78, CloudLinux 7.9, cPanel 110, 1 TB xvda2, 480 Mbps cap. Now 78% (216 GB free).
• IX — ix.azcomputerguru.com, int 172.16.3.10 / ext 72.194.62.5, cPanel 136, /home 4.4 TB (2.7 TB free).
• Jupiter — 172.16.3.20, Unraid, backups share /mnt/user/Backups/websvr-archives/ (97 TB, 31 TB free).
• Neptune Exchange — neptune.acghosting.com, int 172.16.3.11 / ext 67.206.163.124, Exchange 2016, at Dataforth D2; WinRM needs VPN; RMM agent b3a9b454-86eb-491c-ac67-c1f98987d8dc (online). Hosts amtransit.com (6 mailboxes).
• DNS cluster (all role=sync): websvr, IX (72.194.62.5), ns2.acghosting.com=52.52.94.202 (AWS), 184.187.220.69. Our cluster NS: ns1.acghosting.com / ns2.acghosting.com / ns1/ns2.azcomputerguru.com.
• Ralph's Live Transfer session: websvracghosticopya20260717232750wf4 (IX←websvr, running).

Commands & Outputs

• Zone dump/restore-ready: /var/named/<domain>.db per account; backups in repo + ix-copies/.
• Cluster mode: cat /var/cpanel/cluster/root/config/*-dnsrole → sync.
• keepdns removal: whmapi1 removeacct user=<u> keepdns=1.
• Archive: /scripts/pkgacct <u> → /home/cpmove-<u>.tar.gz; Jupiter pull: rsync -a -e "ssh -i /root/.ssh/id_ed25519" root@162.248.93.81:/home/cpmove-<u>.tar.gz /mnt/user/Backups/websvr-archives/.
• Restore any archive: /scripts/restorepkg cpmove-<u> (drop tarball in /home first).
• Transfer status: /usr/local/cpanel/bin/view_transfer websvracghosticopya20260717232750wf4.
• amtransit AD lookup (RMM cmd 94b9a1a6-… on NEPTUNE): ([adsisearcher]"(proxyAddresses=*@amtransit.com*)").FindAll() → 6 mailboxes: driver@, driver-ext@, bertie@ (Ana Lozano), matilde@ (Matilde Slate), nydia@, caitlin@.
• Bandwidth (transfer live): websvr eth0 TX ~434 Mbps; IX eno3 RX ~444 Mbps.

Pending / Incomplete Tasks

• Ralph's Live Transfer in progress (session …232750wf4) — background monitor bdam0du3c running (session-state based; pings on completion/fail/stall/websvr-low). On completion: DNS cutover — repoint ralphstransfer.com A/MX to IX (our cluster). ralphsmoving.com web is on client Cloudflare (separate).
• Furrier moves parked to next month: migrate upr.com (furrier) + desertrat (desertra) websvr→IX under the furrier reseller; fix upr.com's stray owner tag during the move.
• amtransit [BILLING]: 6 mailboxes exist vs 5 billed "Exchange Hosted Email" — reconcile (Winter).
• BirthBiologic SharePoint remediation (Syncro #32187) awaiting Annise's coordinator list; tenant default-link change + Donor Services cleanup staged, not executed.
• mike@ break-glass: rotate sysadmin@azcomputerguru.onmicrosoft.com password in Entra portal (Graph 403 by design), then vault at msp-tools/acg-m365-breakglass.sops.yaml; also exclude from admin-MFA policy already done.
• Reliant Google Business Profile: none exists — needs creation/verification before a reviews link/QR is possible.
• Wiki: run /wiki-compile client:internal-infrastructure --full and /wiki-compile client:arizona-medical-transit --full to fold in this session.

Reference Information

• Ralph's transfer session id: websvracghosticopya20260717232750wf4
• NEPTUNE RMM agent: b3a9b454-86eb-491c-ac67-c1f98987d8dc; amtransit query cmd 94b9a1a6-6148-4331-afe4-7423a20486dd
• Khalsa RMM: client ae08187d-bd2b-43d7-83ac-d82e4fadd6b2, site 11ef9472-1405-4767-b095-7cf2dff0bc3c, code LOWER-TIGER-5022
• BB SharePoint: Syncro #32187 (id 109277420), comment id 424114287; Donor Services site /sites/donorservices, group 11d28c77-c259-4429-8aab-a3a3945f58fd
• Jupiter archives: /mnt/user/Backups/websvr-archives/ — compoundfitness (befef13f…), grabbanddurando (f9a06482…), tucsonsafety (b78b0799…), devgrabblaw (92641a2f…)
• Related log: clients/azcomputerguru.com/session-logs/2026-07-16-mike-mailbox-security-investigation-exchange-tier-breakglass.md