From 10d14719c3bc0ec41389a5723927c9721f9cf6c6 Mon Sep 17 00:00:00 2001 From: Mike Swanson Date: Fri, 17 Jul 2026 17:18:25 -0700 Subject: [PATCH] sync: auto-sync from GURU-5070 at 2026-07-17 17:17:55 Author: Mike Swanson Machine: GURU-5070 Timestamp: 2026-07-17 17:17:55 --- ...ike-websvr-ix-hosting-cleanup-migration.md | 84 +++++++++++++++++++ wiki/clients/arizona-medical-transit.md | 16 +++- 2 files changed, 97 insertions(+), 3 deletions(-) create mode 100644 clients/internal-infrastructure/session-logs/2026-07/2026-07-17-mike-websvr-ix-hosting-cleanup-migration.md diff --git a/clients/internal-infrastructure/session-logs/2026-07/2026-07-17-mike-websvr-ix-hosting-cleanup-migration.md b/clients/internal-infrastructure/session-logs/2026-07/2026-07-17-mike-websvr-ix-hosting-cleanup-migration.md new file mode 100644 index 00000000..ef6d3e78 --- /dev/null +++ b/clients/internal-infrastructure/session-logs/2026-07/2026-07-17-mike-websvr-ix-hosting-cleanup-migration.md @@ -0,0 +1,84 @@ +# websvr→IX hosting cleanup, DNS-safe account removals, furrier reseller, Ralph's Live Transfer, amtransit/Neptune + +## User +- **User:** Mike Swanson (mike) +- **Machine:** GURU-5070 +- **Role:** admin + +## Session Summary + +Large multi-front session dominated by ACG legacy-hosting cleanup and migration on **websvr.acghosting.com** (CloudLinux 7.9, cPanel 110) toward **IX** (ix.azcomputerguru.com, CloudLinux 9, cPanel 136). websvr was at 96% disk (42 GB free); root cause is Ralph's Transfer's 542 GB mailbox account. First trimmed the ext4 root reserve from 5%→1% (`tune2fs -m 1 /dev/xvda2`, +~40 GB usable), then ran a DNS-safe cleanup: scanned all 239 domains across both servers, resolved public A/MX for 194, classified each by account-server vs DNS location, and cross-checked with per-domain Apache traffic logs to confirm which sites each server actually serves. + +Removed **18 stale duplicate accounts** from websvr (accounts whose live copy is on IX/M365/external, zero web traffic on websvr, no live mail there). All removals used `whmapi1 removeacct user= keepdns=1` — critical because IX+websvr share a **cPanel DNS cluster in "sync" mode**, so a normal removeacct's zone deletion would propagate cluster-wide and take domains dark. `keepdns=1` skips the DNS kill entirely (validated on a controlled test with atlantisbevcollc.com: account removed, zones on IX/AWS/public untouched). All 20 affected zones were backed up (websvr + IX copies) to the repo first. + +For sole-copy accounts, established an **archive-then-remove** pattern: `pkgacct` (files+mail+DBs) → **Jupiter** (Unraid, 172.16.3.20) pulls the tarball into `/mnt/user/Backups/websvr-archives/` via temporary SSH key → verify md5 byte-identical on both ends + gzip integrity → `removeacct keepdns=1`. Applied to **compoundfitness** (dead domain, 1.48 GB archive), **grabbanddurando** (12 GB gdapp/data.grabblaw.com stale copy + 29 DBs, 8.9 GB archive), **tucsonsafety** (decommissioned client, 3.0 GB) and **devgrabblaw** (dev site, 9.8 GB). websvr went **96%→78%** (216 GB free). + +Created the **`furrier` reseller on IX** (`furrier.acghosting.com`, standard self-manage ACLs, password `Palo3705S`, vaulted) to fix 7 Furrier-family accounts (westernt, jackfurr, jefffurr, sparcora, thecentu, ultraper, upracing) that were orphan-owned by a non-existent `furrier` owner — now properly attached (8 accounts). Planned Ralph's migration; Mike kicked off a **cPanel Live Transfer** (IX pulls ralphstransfer from websvr, session `websvracghosticopya20260717232750wf4`) which is running (~434 Mbps ≈ websvr's 480 Mbps cap, ETA ~7:15–7:30 PM). Finally, located **amtransit.com** mail (not on IX/websvr) on **Neptune Exchange** — confirmed via a GuruRMM on-box AD `proxyAddresses` lookup (Neptune WinRM needs VPN), returning 6 mailboxes, and updated the amtransit wiki. + +Earlier in the day (separate committed log `clients/azcomputerguru.com/session-logs/2026-07-16-mike-mailbox-security-investigation-exchange-tier-breakglass.md`): mike@ mailbox security investigation, the Exchange-tier (`exchange-op` vs `investigator-exo`) fix, WHfB pruning, and break-glass CA exclusions. Also this session: Khalsa Montessori RMM onboarding (17 SC→RMM), BirthBiologic SharePoint permissions audit → Syncro #32187 comment, Apple supervision cert vaulted, Reliant Google-Business-Profile finding (none exists), and the desertrat/Furrier login incident (server healthy; recovered). + +## Key Decisions +- **`removeacct keepdns=1` instead of editing DNS cluster roles.** Cleaner than temporarily setting websvr's cluster role to standalone — same outcome (no zone-delete propagation), zero risk to the live cluster config that serves all domains. Validated on one account before batching. +- **Archive to Jupiter via Jupiter-pull, not a 500 GB tarball or through the workstation.** pkgacct + Jupiter rsync-pull (temp key) keeps data server-to-server; md5-verified both ends before any removal. +- **Backed up all zones (both servers) before removals** even though keepdns=1 preserves them — safety net, committed to repo. +- **tucsonsafety removal gated on explicit confirmation** — it's a live client with external DNS (Network Solutions) we can't redirect; only removed after Mike confirmed decommission. +- **furrier reseller main domain = furrier.acghosting.com** (controlled placeholder), standard self-manage ACLs — upr.com becomes its real domain when it migrates next month. +- **Ralph's via cPanel Live Transfer (Mike's call)** — native sync-during-copy beats manual config-first+rsync; DNS cutover clean since we control ralphstransfer.com (our cluster). +- **amtransit via RMM on-box AD lookup** (Mike's suggestion) — bypassed the Neptune VPN/WinRM barrier; SYSTEM/computer-account can read AD proxyAddresses without Exchange RBAC. + +## Problems Encountered +- **`view_transfer` path mangled by Git-Bash MSYS** (`C:/Program: No such file`) — `/usr/local/...` args auto-converted. Fixed by base64-encoding remote commands (`echo | base64 -d | bash`). +- **pkgacct "completion" false-positives** — my wait loops matched the wrong process-name pattern (actual proc is `pkgacct - - av: 4 - ...` with dashes) and sampled mid-write tarballs (bogus 149 MB / gzip-fail). Fixed pattern; verified via stable size + gzip -t. +- **Ralph's transfer "PID gone" false alarm** — cPanel transfers re-parent between copy and restore phases, so the initial master PID exits. Replaced the PID-keyed monitor with a session-state monitor (`view_transfer` "is running" header). +- **Long-held SSH monitor session dropped** ("Network error: Software caused connection abort"). Rebuilt monitors to use short per-cycle connections instead of one long-held session. +- **Neptune unreachable via WinRM from GURU-5070** (port 5985 closed; needs VPN). Worked around via GuruRMM agent on-box. +- **grabbanddurando initially mis-diagnosed** as "broken on IX / incomplete migration" — I checked a stale docroot (`/public_html/data`, empty). The real app is at `/public_html/data_grabbanddurando/gdapp` (16 GB) with all DBs; data.grabblaw.com serves 200 on IX. Corrected. +- **Vault redact filter leaked a SA private key** to transcript (grep denylist missed `private_key`). Logged as friction; noted allowlist approach. + +## Configuration Changes +- websvr `/dev/xvda2`: ext4 reserved-blocks 5%→1% (`tune2fs -m 1`), +~40 GB usable. +- websvr accounts removed (all keepdns=1): atlantisbevco, gurushow, danaise, farwest, jrkco, bruceext, peacefulspirit, grabblaw, arizonahatters, desertfox, fsgtucson, nwpool, cryoweave, antivaxxer, westernt, azcomputerguru, azrestaurant, outaboundssports, compoundfitness, grabbanddurando, tucsonsafety, devgrabblaw (22 total accounts removed across the session; 18 stale-duplicate + 4 archived). +- IX: created `furrier` reseller (createacct + setupreseller makeowner=1 + setacls standard). +- Repo: `.claude/skills/remediation-tool/SKILL.md`, `references/gotchas.md`, `scripts/user-breach-check.sh` (Exchange-tier fix); `scripts/breakglass-signin-alert.sh` + `run-breakglass-alert.cmd` (new); `.claude/memory/reference_exchange_online_exchangeop_tier.md` (new) + MEMORY.md; `clients/internal-infrastructure/dns-zone-backups/2026-07-17-websvr-safe-cleanup/` (40 zone files + manifest); `wiki/clients/arizona-medical-transit.md` (Neptune + mailboxes); root `.gitignore` (scratch filters) + swept ~5.3 MB tracked root scratch. +- GURU-5070: scheduled task `ACG-BreakGlass-SignIn-Alert` (every 4h). + +## Credentials & Secrets +- **furrier reseller (IX):** user `furrier`, password `Palo3705S`, WHM https://ix.azcomputerguru.com:2087 — vaulted `clients/furrier/ix-cpanel-reseller.sops.yaml`. +- **Apple supervision cert:** passphrase `caCNnQBVbq` + base64 of SupervisionIdentityCertificate.p12 (CN=mdm_supervision_identity) — vaulted `msp-tools/apple-device-pairing-cert.sops.yaml`. +- **Khalsa RMM site key:** site_code `LOWER-TIGER-5022` — vaulted `clients/khalsa/gururmm-site-main.sops.yaml`. +- websvr root, IX (key `~/.ssh/id_ed25519`), Jupiter (key), Neptune (`ACG\administrator`, WinRM/VPN) — existing vault entries. + +## Infrastructure & Servers +- **websvr.acghosting.com** — 162.248.93.81/.233/.78, CloudLinux 7.9, cPanel 110, 1 TB xvda2, 480 Mbps cap. Now 78% (216 GB free). +- **IX** — ix.azcomputerguru.com, int 172.16.3.10 / ext 72.194.62.5, cPanel 136, /home 4.4 TB (2.7 TB free). +- **Jupiter** — 172.16.3.20, Unraid, backups share `/mnt/user/Backups/websvr-archives/` (97 TB, 31 TB free). +- **Neptune Exchange** — neptune.acghosting.com, int 172.16.3.11 / ext 67.206.163.124, Exchange 2016, at Dataforth D2; WinRM needs VPN; RMM agent `b3a9b454-86eb-491c-ac67-c1f98987d8dc` (online). Hosts amtransit.com (6 mailboxes). +- **DNS cluster** (all role=`sync`): websvr, IX (72.194.62.5), ns2.acghosting.com=52.52.94.202 (AWS), 184.187.220.69. Our cluster NS: ns1.acghosting.com / ns2.acghosting.com / ns1/ns2.azcomputerguru.com. +- **Ralph's Live Transfer session:** `websvracghosticopya20260717232750wf4` (IX←websvr, running). + +## Commands & Outputs +- Zone dump/restore-ready: `/var/named/.db` per account; backups in repo + `ix-copies/`. +- Cluster mode: `cat /var/cpanel/cluster/root/config/*-dnsrole` → `sync`. +- keepdns removal: `whmapi1 removeacct user= keepdns=1`. +- Archive: `/scripts/pkgacct ` → `/home/cpmove-.tar.gz`; Jupiter pull: `rsync -a -e "ssh -i /root/.ssh/id_ed25519" root@162.248.93.81:/home/cpmove-.tar.gz /mnt/user/Backups/websvr-archives/`. +- Restore any archive: `/scripts/restorepkg cpmove-` (drop tarball in /home first). +- Transfer status: `/usr/local/cpanel/bin/view_transfer websvracghosticopya20260717232750wf4`. +- amtransit AD lookup (RMM cmd `94b9a1a6-…` on NEPTUNE): `([adsisearcher]"(proxyAddresses=*@amtransit.com*)").FindAll()` → 6 mailboxes: driver@, driver-ext@, bertie@ (Ana Lozano), matilde@ (Matilde Slate), nydia@, caitlin@. +- Bandwidth (transfer live): websvr eth0 TX ~434 Mbps; IX eno3 RX ~444 Mbps. + +## Pending / Incomplete Tasks +- **Ralph's Live Transfer in progress** (session `…232750wf4`) — background monitor `bdam0du3c` running (session-state based; pings on completion/fail/stall/websvr-low). On completion: DNS cutover — repoint ralphstransfer.com A/MX to IX (our cluster). ralphsmoving.com web is on client Cloudflare (separate). +- **Furrier moves parked to next month:** migrate upr.com (`furrier`) + desertrat (`desertra`) websvr→IX under the furrier reseller; fix upr.com's stray owner tag during the move. +- **amtransit [BILLING]:** 6 mailboxes exist vs 5 billed "Exchange Hosted Email" — reconcile (Winter). +- **BirthBiologic SharePoint remediation** (Syncro #32187) awaiting Annise's coordinator list; tenant default-link change + Donor Services cleanup staged, not executed. +- **mike@ break-glass:** rotate sysadmin@azcomputerguru.onmicrosoft.com password in Entra portal (Graph 403 by design), then vault at `msp-tools/acg-m365-breakglass.sops.yaml`; also exclude from admin-MFA policy already done. +- **Reliant Google Business Profile:** none exists — needs creation/verification before a reviews link/QR is possible. +- **Wiki:** run `/wiki-compile client:internal-infrastructure --full` and `/wiki-compile client:arizona-medical-transit --full` to fold in this session. + +## Reference Information +- Ralph's transfer session id: `websvracghosticopya20260717232750wf4` +- NEPTUNE RMM agent: `b3a9b454-86eb-491c-ac67-c1f98987d8dc`; amtransit query cmd `94b9a1a6-6148-4331-afe4-7423a20486dd` +- Khalsa RMM: client `ae08187d-bd2b-43d7-83ac-d82e4fadd6b2`, site `11ef9472-1405-4767-b095-7cf2dff0bc3c`, code LOWER-TIGER-5022 +- BB SharePoint: Syncro #32187 (id 109277420), comment id 424114287; Donor Services site /sites/donorservices, group `11d28c77-c259-4429-8aab-a3a3945f58fd` +- Jupiter archives: /mnt/user/Backups/websvr-archives/ — compoundfitness (befef13f…), grabbanddurando (f9a06482…), tucsonsafety (b78b0799…), devgrabblaw (92641a2f…) +- Related log: clients/azcomputerguru.com/session-logs/2026-07-16-mike-mailbox-security-investigation-exchange-tier-breakglass.md diff --git a/wiki/clients/arizona-medical-transit.md b/wiki/clients/arizona-medical-transit.md index 9375eaea..8fdb5ddf 100644 --- a/wiki/clients/arizona-medical-transit.md +++ b/wiki/clients/arizona-medical-transit.md @@ -47,8 +47,17 @@ audit; several fields are verified-from-systems, others flagged as gaps to fill. - **Backup:** billed 1x Service - Data Backup, but **NO Backblaze B2 bucket exists for AMT** — backup destination unconfirmed (Datto? another target? not running?). VERIFY. [FLAG] - **AV:** none billed. (Not in scope; small client.) -- **Email:** 5x "Exchange Hosted Email" + spam filtering — **host not documented** (ACG-hosted - Exchange vs third party). Document the actual mail host + admin portal. [GAP] +- **Email:** 5x "Exchange Hosted Email" + spam filtering. **Host = ACG's Neptune Exchange** + (Exchange 2016 — `neptune.acghosting.com` / internal `172.16.3.11` / ext `67.206.163.124`; + physically at Dataforth D2, see [[internal-infrastructure]]). Inbound is MX-routed through + **Mailprotector CloudFilter** (`amtransit-com.inbound.emailservice.io`) then delivered to + Neptune. `amtransit.com` DNS is **external** (ns1/ns2.server312.com — NOT ours). OWA: + `https://neptune.acghosting.com/owa/`; admin via WinRM/EMS as `ACG\administrator` (vault + `clients/dataforth/neptune-exchange.sops.yaml`, needs VPN). Confirmed 2026-07-17 via **GuruRMM** + (AD `proxyAddresses` lookup run on-box on NEPTUNE agent `b3a9b454-…`, since WinRM needs VPN). + NOT on IX or websvr. **6 mailboxes** on amtransit.com: `driver@` (AMT Driver Mail), `driver-ext@` + (Driver Mail), `bertie@` (Ana Lozano), `matilde@` (Matilde Slate), `nydia@`, `caitlin@`. + **[FLAG] 6 mailboxes vs 5 billed** "Exchange Hosted Email" — reconcile with billing (Winter). - **VoIP:** PacketDial / OITVOIP — VOIP 500 + 3 extensions + E911. Manage via `/packetdial`. ## Access @@ -58,7 +67,8 @@ audit; several fields are verified-from-systems, others flagged as gaps to fill. ## Open items (from GPS->RMM audit) 1. Confirm backup destination for the billed Data Backup line (no B2 bucket found). [FLAG — Winter/billing] -2. Document email host + admin portal for the 5 hosted mailboxes. +2. ~~Document email host + mailboxes~~ — **DONE 2026-07-17**: on **Neptune Exchange** (via Mailprotector filter); 6 mailboxes captured via RMM/AD lookup (see Services → Email). +5. **[BILLING] 6 amtransit mailboxes exist vs 5 billed** "Exchange Hosted Email" — reconcile (Winter). 3. Capture key contact name(s) + primary admin credentials -> vault. 4. Reconcile 2 RMM agents vs 1 GPS device billed.