azcomputerguru/claudetools
1
0
Fork 0
Code Issues 73 Pull Requests Actions Packages Projects Releases Wiki Activity

sync: auto-sync from GURU-5070 at 2026-07-17 17:17:55

Browse Source 
Author: Mike Swanson
Machine: GURU-5070
Timestamp: 2026-07-17 17:17:55
This commit is contained in:
Mike Swanson
2026-07-17 17:18:25 -07:00
parent 5cf263fc42
commit 10d14719c3
2 changed files with 97 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

84
clients/internal-infrastructure/session-logs/2026-07/2026-07-17-mike-websvr-ix-hosting-cleanup-migration.md Normal file
View File

@@ -0,0 +1,84 @@
# websvr→IX hosting cleanup, DNS-safe account removals, furrier reseller, Ralph's Live Transfer, amtransit/Neptune
## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin
## Session Summary
Large multi-front session dominated by ACG legacy-hosting cleanup and migration on **websvr.acghosting.com** (CloudLinux 7.9, cPanel 110) toward **IX** (ix.azcomputerguru.com, CloudLinux 9, cPanel 136). websvr was at 96% disk (42 GB free); root cause is Ralph's Transfer's 542 GB mailbox account. First trimmed the ext4 root reserve from 5%→1% (`tune2fs -m 1 /dev/xvda2`, +~40 GB usable), then ran a DNS-safe cleanup: scanned all 239 domains across both servers, resolved public A/MX for 194, classified each by account-server vs DNS location, and cross-checked with per-domain Apache traffic logs to confirm which sites each server actually serves.
Removed **18 stale duplicate accounts** from websvr (accounts whose live copy is on IX/M365/external, zero web traffic on websvr, no live mail there). All removals used `whmapi1 removeacct user=<u> keepdns=1` — critical because IX+websvr share a **cPanel DNS cluster in "sync" mode**, so a normal removeacct's zone deletion would propagate cluster-wide and take domains dark. `keepdns=1` skips the DNS kill entirely (validated on a controlled test with atlantisbevcollc.com: account removed, zones on IX/AWS/public untouched). All 20 affected zones were backed up (websvr + IX copies) to the repo first.
For sole-copy accounts, established an **archive-then-remove** pattern: `pkgacct` (files+mail+DBs) → **Jupiter** (Unraid, 172.16.3.20) pulls the tarball into `/mnt/user/Backups/websvr-archives/` via temporary SSH key → verify md5 byte-identical on both ends + gzip integrity → `removeacct keepdns=1`. Applied to **compoundfitness** (dead domain, 1.48 GB archive), **grabbanddurando** (12 GB gdapp/data.grabblaw.com stale copy + 29 DBs, 8.9 GB archive), **tucsonsafety** (decommissioned client, 3.0 GB) and **devgrabblaw** (dev site, 9.8 GB). websvr went **96%→78%** (216 GB free).
Created the **`furrier` reseller on IX** (`furrier.acghosting.com`, standard self-manage ACLs, password `Palo3705S`, vaulted) to fix 7 Furrier-family accounts (westernt, jackfurr, jefffurr, sparcora, thecentu, ultraper, upracing) that were orphan-owned by a non-existent `furrier` owner — now properly attached (8 accounts). Planned Ralph's migration; Mike kicked off a **cPanel Live Transfer** (IX pulls ralphstransfer from websvr, session `websvracghosticopya20260717232750wf4`) which is running (~434 Mbps ≈ websvr's 480 Mbps cap, ETA ~7:15–7:30 PM). Finally, located **amtransit.com** mail (not on IX/websvr) on **Neptune Exchange** — confirmed via a GuruRMM on-box AD `proxyAddresses` lookup (Neptune WinRM needs VPN), returning 6 mailboxes, and updated the amtransit wiki.
Earlier in the day (separate committed log `clients/azcomputerguru.com/session-logs/2026-07-16-mike-mailbox-security-investigation-exchange-tier-breakglass.md`): mike@ mailbox security investigation, the Exchange-tier (`exchange-op` vs `investigator-exo`) fix, WHfB pruning, and break-glass CA exclusions. Also this session: Khalsa Montessori RMM onboarding (17 SC→RMM), BirthBiologic SharePoint permissions audit → Syncro #32187 comment, Apple supervision cert vaulted, Reliant Google-Business-Profile finding (none exists), and the desertrat/Furrier login incident (server healthy; recovered).
## Key Decisions
- **`removeacct keepdns=1` instead of editing DNS cluster roles.** Cleaner than temporarily setting websvr's cluster role to standalone — same outcome (no zone-delete propagation), zero risk to the live cluster config that serves all domains. Validated on one account before batching.
- **Archive to Jupiter via Jupiter-pull, not a 500 GB tarball or through the workstation.** pkgacct + Jupiter rsync-pull (temp key) keeps data server-to-server; md5-verified both ends before any removal.
- **Backed up all zones (both servers) before removals** even though keepdns=1 preserves them — safety net, committed to repo.
- **tucsonsafety removal gated on explicit confirmation** — it's a live client with external DNS (Network Solutions) we can't redirect; only removed after Mike confirmed decommission.
- **furrier reseller main domain = furrier.acghosting.com** (controlled placeholder), standard self-manage ACLs — upr.com becomes its real domain when it migrates next month.
- **Ralph's via cPanel Live Transfer (Mike's call)** — native sync-during-copy beats manual config-first+rsync; DNS cutover clean since we control ralphstransfer.com (our cluster).
- **amtransit via RMM on-box AD lookup** (Mike's suggestion) — bypassed the Neptune VPN/WinRM barrier; SYSTEM/computer-account can read AD proxyAddresses without Exchange RBAC.
## Problems Encountered
- **`view_transfer` path mangled by Git-Bash MSYS** (`C:/Program: No such file`) — `/usr/local/...` args auto-converted. Fixed by base64-encoding remote commands (`echo <b64> | base64 -d | bash`).
- **pkgacct "completion" false-positives** — my wait loops matched the wrong process-name pattern (actual proc is `pkgacct - <user> - av: 4 - ...` with dashes) and sampled mid-write tarballs (bogus 149 MB / gzip-fail). Fixed pattern; verified via stable size + gzip -t.
- **Ralph's transfer "PID gone" false alarm** — cPanel transfers re-parent between copy and restore phases, so the initial master PID exits. Replaced the PID-keyed monitor with a session-state monitor (`view_transfer` "is running" header).
- **Long-held SSH monitor session dropped** ("Network error: Software caused connection abort"). Rebuilt monitors to use short per-cycle connections instead of one long-held session.
- **Neptune unreachable via WinRM from GURU-5070** (port 5985 closed; needs VPN). Worked around via GuruRMM agent on-box.
- **grabbanddurando initially mis-diagnosed** as "broken on IX / incomplete migration" — I checked a stale docroot (`/public_html/data`, empty). The real app is at `/public_html/data_grabbanddurando/gdapp` (16 GB) with all DBs; data.grabblaw.com serves 200 on IX. Corrected.
- **Vault redact filter leaked a SA private key** to transcript (grep denylist missed `private_key`). Logged as friction; noted allowlist approach.
## Configuration Changes
- websvr `/dev/xvda2`: ext4 reserved-blocks 5%→1% (`tune2fs -m 1`), +~40 GB usable.
- websvr accounts removed (all keepdns=1): atlantisbevco, gurushow, danaise, farwest, jrkco, bruceext, peacefulspirit, grabblaw, arizonahatters, desertfox, fsgtucson, nwpool, cryoweave, antivaxxer, westernt, azcomputerguru, azrestaurant, outaboundssports, compoundfitness, grabbanddurando, tucsonsafety, devgrabblaw (22 total accounts removed across the session; 18 stale-duplicate + 4 archived).
- IX: created `furrier` reseller (createacct + setupreseller makeowner=1 + setacls standard).
- Repo: `.claude/skills/remediation-tool/SKILL.md`, `references/gotchas.md`, `scripts/user-breach-check.sh` (Exchange-tier fix); `scripts/breakglass-signin-alert.sh` + `run-breakglass-alert.cmd` (new); `.claude/memory/reference_exchange_online_exchangeop_tier.md` (new) + MEMORY.md; `clients/internal-infrastructure/dns-zone-backups/2026-07-17-websvr-safe-cleanup/` (40 zone files + manifest); `wiki/clients/arizona-medical-transit.md` (Neptune + mailboxes); root `.gitignore` (scratch filters) + swept ~5.3 MB tracked root scratch.
- GURU-5070: scheduled task `ACG-BreakGlass-SignIn-Alert` (every 4h).
## Credentials & Secrets
- **furrier reseller (IX):** user `furrier`, password `Palo3705S`, WHM https://ix.azcomputerguru.com:2087 — vaulted `clients/furrier/ix-cpanel-reseller.sops.yaml`.
- **Apple supervision cert:** passphrase `caCNnQBVbq` + base64 of SupervisionIdentityCertificate.p12 (CN=mdm_supervision_identity) — vaulted `msp-tools/apple-device-pairing-cert.sops.yaml`.
- **Khalsa RMM site key:** site_code `LOWER-TIGER-5022` — vaulted `clients/khalsa/gururmm-site-main.sops.yaml`.
- websvr root, IX (key `~/.ssh/id_ed25519`), Jupiter (key), Neptune (`ACG\administrator`, WinRM/VPN) — existing vault entries.
## Infrastructure & Servers
- **websvr.acghosting.com** — 162.248.93.81/.233/.78, CloudLinux 7.9, cPanel 110, 1 TB xvda2, 480 Mbps cap. Now 78% (216 GB free).
- **IX** — ix.azcomputerguru.com, int 172.16.3.10 / ext 72.194.62.5, cPanel 136, /home 4.4 TB (2.7 TB free).
- **Jupiter** — 172.16.3.20, Unraid, backups share `/mnt/user/Backups/websvr-archives/` (97 TB, 31 TB free).
- **Neptune Exchange** — neptune.acghosting.com, int 172.16.3.11 / ext 67.206.163.124, Exchange 2016, at Dataforth D2; WinRM needs VPN; RMM agent `b3a9b454-86eb-491c-ac67-c1f98987d8dc` (online). Hosts amtransit.com (6 mailboxes).
- **DNS cluster** (all role=`sync`): websvr, IX (72.194.62.5), ns2.acghosting.com=52.52.94.202 (AWS), 184.187.220.69. Our cluster NS: ns1.acghosting.com / ns2.acghosting.com / ns1/ns2.azcomputerguru.com.
- **Ralph's Live Transfer session:** `websvracghosticopya20260717232750wf4` (IX←websvr, running).
## Commands & Outputs
- Zone dump/restore-ready: `/var/named/<domain>.db` per account; backups in repo + `ix-copies/`.
- Cluster mode: `cat /var/cpanel/cluster/root/config/*-dnsrole` → `sync`.
- keepdns removal: `whmapi1 removeacct user=<u> keepdns=1`.
- Archive: `/scripts/pkgacct <u>` → `/home/cpmove-<u>.tar.gz`; Jupiter pull: `rsync -a -e "ssh -i /root/.ssh/id_ed25519" root@162.248.93.81:/home/cpmove-<u>.tar.gz /mnt/user/Backups/websvr-archives/`.
- Restore any archive: `/scripts/restorepkg cpmove-<u>` (drop tarball in /home first).
- Transfer status: `/usr/local/cpanel/bin/view_transfer websvracghosticopya20260717232750wf4`.
- amtransit AD lookup (RMM cmd `94b9a1a6-…` on NEPTUNE): `([adsisearcher]"(proxyAddresses=*@amtransit.com*)").FindAll()` → 6 mailboxes: driver@, driver-ext@, bertie@ (Ana Lozano), matilde@ (Matilde Slate), nydia@, caitlin@.
- Bandwidth (transfer live): websvr eth0 TX ~434 Mbps; IX eno3 RX ~444 Mbps.
## Pending / Incomplete Tasks
- **Ralph's Live Transfer in progress** (session `…232750wf4`) — background monitor `bdam0du3c` running (session-state based; pings on completion/fail/stall/websvr-low). On completion: DNS cutover — repoint ralphstransfer.com A/MX to IX (our cluster). ralphsmoving.com web is on client Cloudflare (separate).
- **Furrier moves parked to next month:** migrate upr.com (`furrier`) + desertrat (`desertra`) websvr→IX under the furrier reseller; fix upr.com's stray owner tag during the move.
- **amtransit [BILLING]:** 6 mailboxes exist vs 5 billed "Exchange Hosted Email" — reconcile (Winter).
- **BirthBiologic SharePoint remediation** (Syncro #32187) awaiting Annise's coordinator list; tenant default-link change + Donor Services cleanup staged, not executed.
- **mike@ break-glass:** rotate sysadmin@azcomputerguru.onmicrosoft.com password in Entra portal (Graph 403 by design), then vault at `msp-tools/acg-m365-breakglass.sops.yaml`; also exclude from admin-MFA policy already done.
- **Reliant Google Business Profile:** none exists — needs creation/verification before a reviews link/QR is possible.
- **Wiki:** run `/wiki-compile client:internal-infrastructure --full` and `/wiki-compile client:arizona-medical-transit --full` to fold in this session.
## Reference Information
- Ralph's transfer session id: `websvracghosticopya20260717232750wf4`
- NEPTUNE RMM agent: `b3a9b454-86eb-491c-ac67-c1f98987d8dc`; amtransit query cmd `94b9a1a6-6148-4331-afe4-7423a20486dd`
- Khalsa RMM: client `ae08187d-bd2b-43d7-83ac-d82e4fadd6b2`, site `11ef9472-1405-4767-b095-7cf2dff0bc3c`, code LOWER-TIGER-5022
- BB SharePoint: Syncro #32187 (id 109277420), comment id 424114287; Donor Services site /sites/donorservices, group `11d28c77-c259-4429-8aab-a3a3945f58fd`
- Jupiter archives: /mnt/user/Backups/websvr-archives/ — compoundfitness (befef13f…), grabbanddurando (f9a06482…), tucsonsafety (b78b0799…), devgrabblaw (92641a2f…)
- Related log: clients/azcomputerguru.com/session-logs/2026-07-16-mike-mailbox-security-investigation-exchange-tier-breakglass.md

16
wiki/clients/arizona-medical-transit.md
View File

@@ -47,8 +47,17 @@ audit; several fields are verified-from-systems, others flagged as gaps to fill.
- **Backup:** billed 1x Service - Data Backup, but **NO Backblaze B2 bucket exists for AMT** - **Backup:** billed 1x Service - Data Backup, but **NO Backblaze B2 bucket exists for AMT**
— backup destination unconfirmed (Datto? another target? not running?). VERIFY. [FLAG] — backup destination unconfirmed (Datto? another target? not running?). VERIFY. [FLAG]
- **AV:** none billed. (Not in scope; small client.) - **AV:** none billed. (Not in scope; small client.)
- **Email:** 5x "Exchange Hosted Email" + spam filtering — **host not documented** (ACG-hosted - **Email:** 5x "Exchange Hosted Email" + spam filtering. **Host = ACG's Neptune Exchange**
Exchange vs third party). Document the actual mail host + admin portal. [GAP] (Exchange 2016 — `neptune.acghosting.com` / internal `172.16.3.11` / ext `67.206.163.124`;
physically at Dataforth D2, see [[internal-infrastructure]]). Inbound is MX-routed through
**Mailprotector CloudFilter** (`amtransit-com.inbound.emailservice.io`) then delivered to
Neptune. `amtransit.com` DNS is **external** (ns1/ns2.server312.com — NOT ours). OWA:
`https://neptune.acghosting.com/owa/`; admin via WinRM/EMS as `ACG\administrator` (vault
`clients/dataforth/neptune-exchange.sops.yaml`, needs VPN). Confirmed 2026-07-17 via **GuruRMM**
(AD `proxyAddresses` lookup run on-box on NEPTUNE agent `b3a9b454-…`, since WinRM needs VPN).
NOT on IX or websvr. **6 mailboxes** on amtransit.com: `driver@` (AMT Driver Mail), `driver-ext@`
(Driver Mail), `bertie@` (Ana Lozano), `matilde@` (Matilde Slate), `nydia@`, `caitlin@`.
**[FLAG] 6 mailboxes vs 5 billed** "Exchange Hosted Email" — reconcile with billing (Winter).
- **VoIP:** PacketDial / OITVOIP — VOIP 500 + 3 extensions + E911. Manage via `/packetdial`. - **VoIP:** PacketDial / OITVOIP — VOIP 500 + 3 extensions + E911. Manage via `/packetdial`.
## Access ## Access
@@ -58,7 +67,8 @@ audit; several fields are verified-from-systems, others flagged as gaps to fill.
## Open items (from GPS->RMM audit) ## Open items (from GPS->RMM audit)
1. Confirm backup destination for the billed Data Backup line (no B2 bucket found). [FLAG — Winter/billing] 1. Confirm backup destination for the billed Data Backup line (no B2 bucket found). [FLAG — Winter/billing]
2. Document email host + admin portal for the 5 hosted mailboxes. 2. ~~Document email host + mailboxes~~ — **DONE 2026-07-17**: on **Neptune Exchange** (via Mailprotector filter); 6 mailboxes captured via RMM/AD lookup (see Services → Email).
5. **[BILLING] 6 amtransit mailboxes exist vs 5 billed** "Exchange Hosted Email" — reconcile (Winter).
3. Capture key contact name(s) + primary admin credentials -> vault. 3. Capture key contact name(s) + primary admin credentials -> vault.
4. Reconcile 2 RMM agents vs 1 GPS device billed. 4. Reconcile 2 RMM agents vs 1 GPS device billed.