Files
claudetools/wiki/clients/internal-infrastructure.md
Mike Swanson 32f64a9561 wiki: seed 9 client articles (internal-infra, peaceful-spirit, cryoweave, glaztech, pavon, grabb-durando, stamback-septic, sombra-residential, birth-biologic)
Notable findings per article:
- internal-infrastructure: Neptune cert expires 2026-05-31, DkimSigner
  disabled (unsigned outbound mail), Cloudflare tunnel on Jupiter
- peaceful-spirit: L2TP/IPsec RRAS VPN; billing/Syncro ID undocumented
- cryoweave: website redesign pending client assets
- glaztech: phishing bypassed MailProtector via secondary MX (fixed);
  no MFA enforcement yet; do not enable Security Defaults yet
- pavon: OwnCloud cron stacking fixed; Nextcloud migration deferred
- grabb-durando: plaintext DB password in README needs vaulting; AI
  demand review app scoped
- stamback-septic: WS2012 EOL server on network
- sombra-residential: Server2013 is actually WS2012 EOL unpatched
- birth-biologic: Datto→SharePoint migration unconfirmed complete

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-24 19:38:50 -07:00

295 lines
23 KiB
Markdown

---
type: client
name: internal-infrastructure
display_name: ACG Internal Infrastructure
last_compiled: 2026-05-24
compiled_by: DESKTOP-0O8A1RL/claude-main
sources:
- clients/internal-infrastructure/PROJECT_STATE.md
- clients/internal-infrastructure/ix-server-issues-2026-01-13.md
- clients/internal-infrastructure/docs/SSH_ACCESS_SETUP.md
- clients/internal-infrastructure/docs/SSH_CONNECTION_INVESTIGATION_REPORT.md
- clients/internal-infrastructure/reports/2026-04-16-howard-breach-check.md
- clients/internal-infrastructure/vendor-tickets/2026-04-13-cox-bgp-cloudflare-routing.md
- clients/internal-infrastructure/session-logs/2026-03-16-ix-account-cleanup.md
- clients/internal-infrastructure/session-logs/2026-03-17-neptune-exchange-cleanup.md
- clients/internal-infrastructure/session-logs/2026-04-11-smart-slider-security-scan.md
- clients/internal-infrastructure/session-logs/2026-04-13-session.md
- clients/internal-infrastructure/session-logs/2026-04-23-neptune-inbound-mail-outage.md
- .claude/memory/infra_office_network.md
- .claude/memory/reference_ix_server_ssh.md
- .claude/memory/project_email_routing_neptune.md
- CONTEXT.md (root)
backlinks:
- systems/jupiter
- systems/neptune
- projects/msp-tools/guru-rmm
---
# ACG Internal Infrastructure
Arizona Computer Guru's own internal systems, treated as a "client" record for work-tracking purposes. This article covers what lives under `clients/internal-infrastructure/` — ad-hoc operational work on ACG's own hosting servers, mail platform, network, and M365 tenant. It is NOT the primary record for GuruRMM development (see `wiki/projects/guru-rmm.md`), ClaudeTools API development (see `CONTEXT.md` root), or ACG office LAN topology (see `wiki/systems/`). The merge of the former `clients/ix-server/` folder into this one happened 2026-04-13.
---
## Profile
- **Contract type:** Internal (no billing) — ACG's own infrastructure. Work is ad-hoc and reactive.
- **Key contacts:**
| Name | Role | Notes |
|---|---|---|
| Mike Swanson (mike) | Owner / admin | Primary operator |
| Howard Enos (howard) | Technician | Full trust — same access as admin |
- **Billing rate:** N/A — internal only
- **M365 tenant:** azcomputerguru.com | Tenant ID: `ce61461e-81a0-4c84-bb4a-7b354a9a356d`
- **Syncro customer ID:** N/A — ACG's own work is not tracked in Syncro
---
## What This Client Record Covers
This folder tracks reactive work on ACG's own:
- **IX web hosting server** (cPanel/WHM, client websites, WordPress maintenance)
- **Neptune Exchange server** (hosted mail for multiple client domains — physically at Dataforth D2)
- **Cloudflare / DNS** (azcomputerguru.com zone, tunnel, BGP issues)
- **ACG M365 tenant** (azcomputerguru.com — breach checks, CA policy hygiene)
- **ACG office LAN** (pfSense, Jupiter Unraid, VMs) — incidental notes; primary docs are in `wiki/systems/`
Work on **GuruRMM** (development, deployment) lives in `projects/msp-tools/guru-rmm/` and root `session-logs/`. Work on **ClaudeTools API** lives in `projects/` and root `CONTEXT.md`.
---
## Infrastructure
### ACG Office LAN
- **Subnet:** 172.16.0.0/22
- **DNS / Router:** pfSense at 172.16.0.1 (SSH port 2248, user admin); handles Unbound DNS and Tailscale subnet routing
- **Tailscale node:** pfsense-2 (100.119.153.74)
- **Vault:** `infrastructure/pfsense-firewall.sops.yaml`
| Host | IP | Role | Notes |
|---|---|---|---|
| Jupiter | 172.16.3.20 | Unraid NAS — all VMs + Docker | SSH port 22, root. NPM, Gitea, Seafile, GuruRMM VM, cloudflared |
| GuruRMM VM | 172.16.3.30 | Linux VM on Jupiter | GuruRMM server, ClaudeTools API, MariaDB, Coord API |
| Pluto | 172.16.3.36 | Windows Server 2019 VM on Jupiter | MSI build server for GuruRMM agents |
| Uranus | 172.16.3.21 | OwnCloud additional storage | NOT a proxy |
| IX Web Server | 172.16.3.10 | cPanel/WHM web hosting | 87 WordPress sites, CloudLinux 9.7 |
| Neptune Exchange | 172.16.3.11 | Exchange Server 2016 | Physically at Dataforth D2 — NOT ACG office LAN |
| ACG-DC16 | 172.16.3.52 / 172.16.3.50 | Windows Server 2016 DC | AD, DNS for acg.local; all FSMO roles |
### IX Web Hosting Server
- **Hostname:** ix.azcomputerguru.com
- **Internal IP:** 172.16.3.10
- **External IP:** 72.194.62.5
- **OS:** CloudLinux 9.7 (RHEL 9 family)
- **Stack:** Apache, WHM/cPanel, MySQL/MariaDB per-account
- **Sites:** 87 WordPress installations (as of 2026-04-11 scan); 82 cPanel accounts audited 2026-03-16 (14 removed, 7 restored)
- **WHM:** `https://ix.azcomputerguru.com:2087` — must be **DNS-only / grey-cloud** in Cloudflare (port 2087/2083 require direct IP routing; Cloudflare tunnel cannot forward non-standard ports)
- **SSH:** `ssh root@172.16.3.10` (internal) or `ssh root@72.194.62.5` (external)
- **Vault:** `infrastructure/ix-server.sops.yaml`
- **[WARNING] SSH key auth not set up from CachyOS workstation (acg-guru-5070)** — must use sshpass with password from vault when connecting from that machine.
**ACG infrastructure DNS zones on IX** (must never remove the `acg` cPanel account):
- acghosting.com, ns1.acghosting.com, ns2.acghosting.com, fsusa.acghosting.com, websvr.acghosting.com
**Clients with active mail on IX** (accounts kept for non-web services):
- `cascades` — cascadestucson.com (active local mail, populated mailboxes)
- `rrspc` — rrspc.com (active local mail, MX to mail.rrspc.com on IX)
- `glaztech` — glaztech.com (DNS-only account)
- `rarengineer` — rarengineer.com (MX may resolve to IX)
- `thegirlsestate` — thegirlsestatesales.com (mail service account)
### Neptune Exchange Server
Neptune is ACG's on-premises Exchange Server 2016, hosting mail for multiple client domains. It is physically colocated at Dataforth's D2 facility but operates as ACG infrastructure.
- **Hostname:** neptune.acghosting.com / mail.acghosting.com / NEPTUNE.acg.local
- **Internal IP:** 172.16.3.11 (172.16.x.x subnet — NOT at ACG office despite the IP)
- **External IP:** 67.206.163.124 (inbound); 67.206.163.122 (outbound)
- **OS:** [WARNING] Windows Server 2022 (in-place upgraded from WS2016 on 2026-04-22 — Exchange 2016 is UNSUPPORTED on WS2022)
- **Exchange:** 2016 Standard Evaluation, Build 15.1.2507.17
- **AD Domain:** acg.local
- **DNS Server (primary):** ACG-DC16 at 172.16.3.52 (also .50)
- **Mailboxes:** 56 total (N-Hosting1 DB: 809 GB / 54 boxes; N-LargeBoxes DB: 313 GB / 2 boxes)
- **Let's Encrypt cert:** CN=mail.acghosting.com, expires 2026-05-31 [WARNING] — renewal needed
- **Internal transport cert:** Thumbprint `E58BFCBAEFEFDCAED0BF9E894127A3DE64CE9C69`, expires 2026-07-22 [WARNING]
- **Access:** Local PowerShell with Exchange Management Shell snapin (`Add-PSSnapin Microsoft.Exchange.Management.PowerShell.SnapIn`); must run as administrator.ACG on the box or via domain-admin WinRM
- **Vault:** `infrastructure/neptune-exchange.sops.yaml` [unverified — check vault for current entry name]
**Accepted domains on Neptune** (19 client-hosted):
acg.local, acghosting.com (ExternalRelay), airandspaceacademy.com, amtransit.com, devconllc.com, farwestwell.com, goldenchoicecatering.com, jparkinsonaz.com, justsimplysmart.com, lifelonglearningacademy.com, littleheartslittlehands.com, littleheartslittlehands.org, outaboundssports.com, packetdial.com, patriotinternalmedicine.com, rieussetcorp.com, simplehost.email (Default), tucsongoldencorral.com, tucsonsafety.com
**Outbound SBR send connectors** (via Mailprotector / emailservice.io smarthosts): devconllc, littleheartslittlehands/airandspaceacademy, patriotinternalmedicine, farwestwell, tucsongoldencorral, lifelonglearningacademy, amtransit, tucsonsafety, rieussetcorp/Sorensen, horseshoemgt, catch-all (DNS)
**DKIM signing** (Exchange DkimSigner — currently DISABLED after 2026-04-23 KB outage): amtransit.com (s1), littleheartslittlehands.org (default), tucsongoldencorral.com (dkim), devconllc.com (default), jparkinsonaz.com (s1), rieussetcorp.com (s1). Keys in `C:\Program Files\Exchange DkimSigner\keys\`
**Transport rules** (3): Restrict Inbound - Devcon and LittleHearts (priority 0), Webhost Spam (priority 1), Bardach BCC (priority 2)
**[WARNING] Critical post-WS2022-upgrade changes that must survive reboots** (applied 2026-04-23):
- `Set-TransportServer NEPTUNE -InternalDNSAdapterEnabled $false -InternalDNSServers @('172.16.3.50','172.16.3.52')` — Exchange transport DNS must NOT use adapter-mode on WS2022 (edgetransport bypasses suffix search list; causes `DnsDomainDoesNotExist` for short names like n-hosting1)
- `Exchange DkimSigner` transport agent: DISABLED (went async on OnCategorizedMessage after .NET CU)
- `messageconcept SenderBasedRouting` transport agent: DISABLED (expired license; MS SBR at priority 12 handles outbound routing)
- IRM fully disabled: `Set-IRMConfiguration -InternalLicensingEnabled $false -ExternalLicensingEnabled $false -TransportDecryptionSetting Disabled ...`
- `HKLM\SYSTEM\CurrentControlSet\Services\AssistantsQuarantine` ACL: NETWORK SERVICE has FullControl (inheritable) — added because WS2022 default ACL excludes NETWORK SERVICE, causing Event 10003 delivery crashes
- DC-side DNS A records on ACG-DC16: n-hosting1 → 172.16.3.11, n-largeboxes → 172.16.3.11, mail.acg.local → 172.16.3.11
- Hosts file on Neptune: MAIL → 172.16.3.11, mail.acg.local → 172.16.3.11, n-hosting1 → 172.16.3.11, n-largeboxes → 172.16.3.11 (belt-and-suspenders; edgetransport bypasses hosts file but other processes use it)
- `msExchRoutingMasterDN` set to NEPTUNE DN (was pointing to tombstoned MAIL server AD object)
- MSExchangeADTopology: 45-sec SCM start timeout on every cold boot on WS2022 — manual `Start-Service MSExchangeADTopology` then start remaining services in dependency order is required after every reboot
**Dead MAIL server AD carcass** (still in AD — decommission pending):
- `CN=MAIL,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),...`
- Has 6 attached receive connectors and the WesternTire Relay connector — all can be removed with the server object
- Must remove via ADSI Edit (`Remove-ADObject -Recursive`) — no physical server exists
**Migration plan** (decided 2026-04-23): Build fresh WS2022 VM, install Exchange 2019 CU14+ (supported OS combo), move 56 mailboxes, repoint MailProtector relay + public DNS + AutoDiscover, force-remove both NEPTUNE and MAIL carcasses. Full runbook at `C:\NeptuneConfigExport-20260423\MIGRATION-RUNBOOK.md` on NEPTUNE — copy this folder before NEPTUNE goes away. Do NOT run `/PrepareSchema` without a system-state backup of ACG-DC16 first (single-DC forest; schema changes are forest-permanent).
### Cloudflare / DNS
- **Zone:** azcomputerguru.com — Zone ID `1beb9917c22b54be32e5215df2c227ce`
- **Account:** Mike@azcomputerguru.com's Account, Pro Website plan
- **CF API tokens:** in 1Password. Vault metadata only at `services/cloudflare.sops.yaml` (tokens not yet migrated to SOPS — pending action from 2026-04-13)
- **Cloudflare Tunnel:** `acg-origin` (UUID `78d3e58f-1979-4f0e-a28b-98d6b3c3d867`) running as Docker container `cloudflared` on Jupiter (`/mnt/cache/appdata/cloudflared/`). Deployed 2026-04-13 as workaround for Cox BGP routing failure.
**Tunneled hostnames** (9, all returning HTTP 200 via tunnel as of 2026-04-13):
- To IX (172.16.3.10:443): azcomputerguru.com, analytics., community., radio.
- To Jupiter NPM (172.16.3.20:18443): git., plexrequest., rmm., rmm-api., sync.
**Grey-clouded (DNS-only) hostnames** — direct to public IP, NOT through tunnel:
- `ix.azcomputerguru.com` → A 72.194.62.5 (must stay grey-cloud; WHM/cPanel on :2087/:2083 require direct routing)
- `rmm-api.azcomputerguru.com` → [WARNING] must stay grey-cloud or DNS-only — Cloudflare proxy blocks WebSockets; GuruRMM agents use WebSocket to rmm-api. See Gitea Issue #9.
**Unresolved / still broken hostnames** (as of 2026-04-13; no user-visible regression but not fixed):
- `plex.azcomputerguru.com` (525) — needs Jupiter NPM vhost for Plex container
- `rustdesk.azcomputerguru.com` (525) — rustdesk server location unknown; may be decommissioned
- `secure.azcomputerguru.com` (ERR) — points to 172.16.1.16 which Jupiter cannot route to
### ACG M365 Tenant
- **Domain:** azcomputerguru.com
- **Tenant ID:** `ce61461e-81a0-4c84-bb4a-7b354a9a356d`
- **MSP multi-tenant app (Claude-MSP-Access):** App ID `fabb3421-8b34-484b-bc17-e46de9703418` — vault: msp-tools SOPS file
---
## Access
| Resource | Method | Notes |
|---|---|---|
| IX (internal) | `ssh root@172.16.3.10` | Vault: `infrastructure/ix-server.sops.yaml` |
| IX (external) | `ssh root@72.194.62.5` | Same credentials |
| IX WHM | `https://ix.azcomputerguru.com:2087` | Must be grey-cloud in CF; NAT via pfSense |
| Jupiter | `ssh root@172.16.3.20` | Vault: `infrastructure/jupiter-unraid-primary.sops.yaml` |
| pfSense | `ssh admin@172.16.0.1 -p 2248` | Vault: `infrastructure/pfsense-firewall.sops.yaml` |
| Neptune | Local PowerShell as administrator.ACG (on-box) | Also: WinRM from ACG-DC16; no WinRM from external without VPN |
| ACG-DC16 | `Invoke-Command -ComputerName ACG-DC16` (from domain-joined box) | Kerberos via SPN-matching hostname required |
| ACG M365 | Graph API via Claude-MSP-Access app | Vault: msp-tools SOPS file |
| Cloudflare API | Bearer token from 1Password | Partial: lacks Zone Settings + Analytics permissions |
**SSH passwordless automation to GuruRMM VM (172.16.3.30):**
RSA 4096-bit key at `C:\Users\MikeSwanson\.ssh\id_rsa`; public key authorized for `guru@172.16.3.30`. See `clients/internal-infrastructure/docs/SSH_ACCESS_SETUP.md`.
---
## Patterns & Known Issues
### IX Web Server — WordPress Hygiene
IX hosts 87 WordPress sites. Recurring issues:
- **Wordfence database bloat** (wp_wffilemods, wp_wfknownfilelist) — present on most sites; needs periodic truncation
- **Error logs growing unchecked** — arizonahatters.com hit 468 MB (2026-01-13). Log rotation via logrotate not yet deployed.
- **WP_DEBUG enabled on production sites** — debug.log files grow unbounded (gentlemansacres.com: 350 MB, azrestaurant.com: 181 MB as of scan)
- **5 critically outdated WordPress sites** (security risk — unaddressed since 2026-03-16 cleanup)
- **Supply chain attack awareness:** Smart Slider 3 Pro supply chain attack (April 7-9, 2026) — IX was not affected (0 Pro installations; 3 Free installations all safe). Scan script at `/root/scan_smart_slider.sh` on IX.
- **Old backups consuming disk:** azcomputerguru (3 GB+), acepickupparts (1.6 GB), sundanzer (2 GB) on IX — not offloaded
### IX cPanel Account Hygiene
Lesson from 2026-03-16 cleanup: DNS migration alone does not mean mail/DNS services have migrated. Always verify non-HTTP services before removing an account. The `acg` account contains critical NS1/NS2 infrastructure DNS zones — never remove it.
### Neptune Exchange — Systemic Fragility
Neptune is Exchange 2016 running on an unsupported OS (WS2022 after the 2026-04-22 in-place upgrade). Three classes of problems recur:
1. **Windows Update / CU-triggered service restarts surface latent issues** — the 2026-04-23 outage involved 4 separate latent problems surfacing simultaneously after KB5082142 + KB5084071 forced transport service reload. After any Exchange or OS CU, verify end-to-end DELIVER (not just SMTP-accept) within 10 minutes.
2. **`MSExchangeADTopology` 45-sec SCM timeout on cold boot** — every reboot on WS2022 requires manual `Start-Service MSExchangeADTopology` first, then starting remaining 25 Exchange services in dependency order. Treat reboots as planned events.
3. **edgetransport internal DNS does not follow suffix search list on WS2022** — short names like `n-hosting1` resolve fine via .NET/OS resolver but fail in Exchange's own DNS client unless explicit DNS servers are set (`Set-TransportServer -InternalDNSAdapterEnabled $false`). DC-side A records AND the explicit DNS server config must both be in place.
**Recurring email routing issues:** Sorensen (rieussetcorp) and devcon have both hit outbound routing failures; when one breaks, check if SBR config applies to the other too. See `memory/project_email_routing_neptune.md`.
**Mailprotector SBR routing:** Two agents on Neptune — `messageconcept ExSBR` (DISABLED, expired license) and `Sender Based Routing` (Microsoft, priority 12, ACTIVE). SBR config files at `C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\agents\Custom\Microsoft.Exchange.SBR.{InternalDomains,OverrideSettings,IgnoreAuthAs}.config`. After any SBR config change: `Restart-Service MSExchangeTransport -Force`.
**Outbound spam / DKIM hygiene:**
- Exchange DkimSigner is DISABLED — outbound mail currently lacks DKIM signatures. Receivers with strict DMARC p=reject (devconllc.com is the one ACG operates) may reject replies. Re-enabling requires verifying DkimSigner is compatible with the post-.NET-CU runtime.
- messageconcept ExSBR can be fully uninstalled (DLL at `C:\Program Files\messageconcept\ExSBR\`, registry key `HKLM\SOFTWARE\SenderBasedRouting`).
**Pending transport cert renewal:** Thumbprint `E58BFCBAEFEFDCAED0BF9E894127A3DE64CE9C69` expires 2026-07-22.
**Pending Neptune Let's Encrypt renewal:** CN=mail.acghosting.com cert expires 2026-05-31 — URGENT.
**Incomplete domain MX fixes from 2026-03-17** (still unresolved as of last session):
- `airandspaceacademy.com`: DNS on GoDaddy still points MX to mail.acghosting.com (direct, no filter) — being rejected by the transport inbound restriction rule. Needs changing to Mailprotector inbound.
- `littleheartslittlehands.com`: DNS on Cloudflare points MX to cbsolt.net — needs Mailprotector.
- `littleheartslittlehands.org` DMARC: still p=none (could tighten to p=reject like devcon).
### Cox BGP Routing Issue
Cox ISP has broken BGP routing from ACG's netblock (72.194.62.0/29) to specific Cloudflare IP prefixes (162.158.0.0/16, 172.64.0.0/13, 173.245.48.0/20, 141.101.64.0/18). Cloudflare tunnel on Jupiter is the workaround. Cox escalation ticket drafted at `clients/internal-infrastructure/vendor-tickets/2026-04-13-cox-bgp-cloudflare-routing.md` — status: [unverified] not confirmed submitted to Cox as of last session.
### ACG M365 Tenant Hygiene
From 2026-04-16 Howard breach check:
- **Active credential-stuffing campaign** against howard@azcomputerguru.com — 174 foreign attempts in 30 days (CN, IN, KR, LU via Azure CLI, BR, DE, JP targeting admin endpoints). All blocked. Pattern indicates attacker knows Howard is an MSP admin and probes Exchange Online basic auth + Azure AD PowerShell.
- **Howard's password was 18 months old** (last changed 2024-09-24) — rotation recommended.
- **Gap: ComputerGuru - AI Remediation SP lacks Exchange Administrator role in our own tenant** — blocks hidden inbox rule checks, delegate audits, mailbox-level forwarding checks. Fix: Entra → Roles → Exchange Administrator → add the app SP.
- **Gap: IdentityRiskyUser.Read.All not consented in azcomputerguru tenant** — blocks Identity Protection checks.
- [unverified] Whether Howard's password was rotated after this check.
### ClaudeTools Hook / SSH Process Accumulation
The Claude Code hooks (user-prompt-submit, task-complete) spawn background `sync-contexts` processes with `&`. Combined with `core.sshcommand = OpenSSH` in git config, this causes SSH processes to accumulate (~1-2 per user message) without cleanup. Investigation report at `clients/internal-infrastructure/docs/SSH_CONNECTION_INVESTIGATION_REPORT.md`. Recommended fix: remove background `&` spawn from hooks or add process cleanup traps. [unverified] Whether this was addressed.
---
## Active Work
As of last session (2026-04-23):
- **Neptune Exchange migration** — Build Exchange 2019 on fresh WS2022 VM. Runbook at `C:\NeptuneConfigExport-20260423\MIGRATION-RUNBOOK.md` on Neptune. Mike building the VM. Critical gate: **back up ACG-DC16 before running `/PrepareSchema`** (forest-permanent, no rollback).
- **Neptune Let's Encrypt cert** — expires 2026-05-31. Renewal critical.
- **Neptune internal transport cert** — expires 2026-07-22.
- **DkimSigner re-enable / replace** — outbound mail currently unsigned. Evaluate whether Exchange DkimSigner is runtime-compatible post-KB5084071, or replace with alternative.
- **MAIL server AD decommission** — once Exchange 2019 is live and mailboxes moved: `Remove-ADObject -Recursive` on the MAIL carcass. After that, remove hosts file entries for MAIL/mail.acg.local and DC-side DNS records (n-hosting1, n-largeboxes, mail can remain or be repurposed for the new server).
- **Cox BGP ticket** — submit if not already done (`vendor-tickets/2026-04-13-cox-bgp-cloudflare-routing.md`).
- **Cloudflare tokens** — migrate from 1Password-only to SOPS vault (`services/cloudflare.sops.yaml`) for pipeline use.
- **IX WordPress hygiene** — 5 critically outdated sites, log rotation, WP_DEBUG on production (low urgency unless a site is actively impacted).
- **plex/rustdesk/secure hostnames** — still returning 5xx/ERR; need NPM vhost additions and/or routing fixes.
---
## History Highlights
| Date | Event |
|---|---|
| 2026-01-13 | IX server critical performance scan — arizonahatters.com 468 MB error log, peacefulspirit 310 MB DB bloat, Wordfence widespread. Documented; cleanup partially executed. |
| 2026-01-17 | SSH process accumulation investigation — hook background-spawn pattern identified as cause. |
| 2026-03-16 | IX account cleanup — 82 cPanel accounts audited, 14 removed, 7 restored. 8.5 GB error logs truncated. 60 inactive plugins removed. 4 WordPress nav-menu.php fatal errors fixed. `clients/ix-server/` folder (later merged into this one). |
| 2026-03-17 | Neptune Exchange cleanup — 9 stale accepted domains removed, 24 mailboxes disabled, send connectors moved from dead MAIL server to NEPTUNE, SBR routing for devcon + littlehearts restored, devconllc.com DMARC tightened to p=reject, 20,473 spam messages purged. |
| 2026-04-11 | IX Smart Slider 3 Pro supply chain attack scan — 87 WP sites scanned; 0 Pro installations; not affected. |
| 2026-04-13 | Cox BGP / Cloudflare 521 incident — broken BGP for CF prefixes 162.158/172.64/173.245/141.101. Cloudflare Tunnel deployed on Jupiter Docker (`acg-origin`). 9 hostnames tunneled. `clients/ix-server/` merged into `clients/internal-infrastructure/`. |
| 2026-04-16 | Howard breach check on azcomputerguru.com M365 — no breach; credential-stuffing campaign active (all blocked); password age 18 months; Exchange Admin role missing from our own tenant for remediation app. |
| 2026-04-22 | Neptune in-place upgraded from WS2016 → WS2022 (unsupported with Exchange 2016). |
| 2026-04-23 | **Neptune mail outage (~42 min)** — triggered by KB5082142 + KB5084071 CUs forcing Exchange service reload after WS2022 upgrade exposed 4 latent incompatibilities: registry ACL crash, dead MAIL server proxy routing, DkimSigner async bug, RMS + Index Routing agent timeouts. 7 fixes applied. Mail restored 14:32. Exchange 2019 migration plan agreed. |
---
## Backlinks
- [[systems/jupiter]] — Unraid primary: hosts GuruRMM VM, NPM, Gitea, cloudflared tunnel, Pluto build server VM
- [[systems/neptune]] — Exchange Server 2016 at Dataforth D2; full article if it exists
- [[wiki/clients/dataforth]] — Neptune physically colocated at Dataforth D2; Neptune's 172.16.x.x IP routes through D2TESTNAS
- [[projects/msp-tools/guru-rmm]] — GuruRMM server runs on ACG office infrastructure (172.16.3.30)