90 lines
13 KiB
Markdown
90 lines
13 KiB
Markdown
## User
|
||
- **User:** Mike Swanson (mike)
|
||
- **Machine:** GURU-5070
|
||
- **Role:** admin
|
||
|
||
## Session Summary
|
||
|
||
A long multi-client MSP session spanning Entra Connect (AAD Connect) remediation, an internet-facing VPN brute-force takedown, new UniFi gateway tooling, a server health/security hardening pass, an EOP mail investigation, and a Syncro invoice-automation build.
|
||
|
||
Started by provisioning a new GuruRMM client "Russo, Steve" (Russo Law Firm) with two sites (Main, Shannon) and vaulting both one-time enrollment keys. The user then reported "Russo is having an ad sync issue." Diagnosis on RUSSO-SRV (rrs-law.com DC) found Entra Connect healthy except one persistent AD-connector export error: writeback of `msDS-KeyCredentialLink` (Windows Hello key) to `guru@rrs-law.com` denied with LDAP 8344, failing since 2025-05-07 (17.7k retries). Root cause: the account is AdminSDHolder-protected (Domain/Enterprise/Schema Admins), so SDProp strips the connector account's write permission. Fixed by delegating `WP;msDS-KeyCredentialLink` on AdminSDHolder via dsacls + forcing SDProp + a delta sync; verified zero export errors. Ticketed/billed #32302 (1 hr warranty, public+emailed resolution).
|
||
|
||
Repeated the same diagnosis on Glaztech (GTI-INV-DC, glaztech.com DC) — identical fault on a different attribute: `msExchSafeSendersHash` (Exchange hybrid writeback) denied on `seastman` since 2025-08-28. Applied the same AdminSDHolder fix. Cloud-side also surfaced a separate UPN PropertyConflict: `CAS@glaztech.com` was both the `CAS` user's UPN and an alias on Alex Miramontez, parking the synced CAS object as `CAS1944@onmicrosoft`. Resolved by changing the CAS account's on-prem UPN to its own primary (`CAS2@glaztech.com`) — conflict cleared, no mail-routing change. Ticketed/billed #32426 (1 hr remote, prepaid).
|
||
|
||
Grabb & Durando: investigating GND-SERVER's "needs a reboot for VPN to work" symptom revealed an internet-facing **PPTP RRAS VPN under sustained brute-force** (~1,900 failed attempts/14 days, 0 legit logins). The edge is a UniFi USG-3P (WAN 174.76.185.203) port-forwarding 80/443/1723 from `src=any` to GND-SERVER (192.168.242.200). To remediate at the gateway, **built a new unifi-wifi skill script `gw-control.sh`** (port-forward + WAN firewall + block-ips actions, gated/DRY-RUN/rollback, same RW-REST pattern as apply-wlan), after pulling Howard's 15 newer unifi commits and coordinating via coord lock+messages. Applied Option A (`pf-set-ports VPN 80,443` to drop 1723 + `fw-disable GRE`), then restarted RRAS to free the wedged ports. Added a pfSense gateway "compatibility layer" roadmap item (§ E). Emergency-ticketed #32427 (0.5 h ×1.5 prepaid).
|
||
|
||
Ran a full onboarding diagnostic on GND-SERVER (graded RED) and found two probe false-positives (Server-2019 flagged EOL via the Win10-1809 build collision; Kernel-Boot event 153 "VBS disabled" miscounted as a disk error) — fixed both in `onboarding-diagnostic.ps1`. Hardened the box: disabled the dormant domain admin `tempadmin` (unused since 2022), rotated the domain `Administrator` password (unchanged since 2019) and vaulted it, and scheduled an off-hours maintenance reboot. Confirmed no workstations are enrolled in Grabb's RMM site. Then checked EOP for `cansley@starrpass.com` (user corrected that starrpass.com is direct-to-Microsoft, not Mailprotector): 0 quarantined, 2 old Apple "rejections" that were `550 5.1.10 RecipientNotFound` from before the mailbox existed. Finally, built a Syncro invoice-note automation: the on-screen "Invoice Message" is the invoice `note` field, now auto-populated with a block-rate upsell hint (non-block customers), an hours-remaining count (block customers), and a low-block (<4 hr) renewal line that tags Winter in #bot-alerts; plus a recurring-invoice sweep.
|
||
|
||
## Key Decisions
|
||
|
||
- **Don't disable the domain built-in Administrator on a DC** — it's the break-glass account; rotated its stale (2019) password and vaulted instead, left enabled. Disabled the genuinely dormant `tempadmin` (reversible).
|
||
- **AdminSDHolder grant, not OU/object grant** — protected accounts have inheritance stripped by SDProp, so the permission must live on AdminSDHolder to persist; `dsacls /G` is additive (one attribute-scoped ACE).
|
||
- **CAS UPN conflict fixed by renaming the CAS account's UPN** (to its own primary CAS2@glaztech.com), not by removing Alex's alias — zero mail-routing change, since CAS@ already routes to Alex.
|
||
- **Grabb VPN: close PPTP at the USG, keep 443/80** — dropped only 1723 + GRE (PPTP needs both), preserving SSTP/web; 0 legit logins in 14 days made it low-risk.
|
||
- **Built gw-control.sh rather than one-off API calls** — the firewall/port-forward gap was the remaining unifi ROADMAP item; coordinated with Howard (who was actively shipping) via lock + messages to avoid collision.
|
||
- **`< 4 hrs` absolute threshold for low-block** (user's simplification) instead of "25% of full block," because the API's `prepay_hours` is only the current balance — full block size isn't exposed.
|
||
- **Recurring invoice notes via a post-run sweep** — Syncro generates recurring invoices server-side, so a dynamic per-remaining-hours note requires an idempotent sweep, not a static schedule note.
|
||
- **Fixed probe false-positives at the source** rather than just flagging — both bugs would mis-grade every Windows Server in the fleet.
|
||
|
||
## Problems Encountered
|
||
|
||
- **AAD Connect run-step detail not exposed by `Get-ADSyncRunProfileResult`** on the older builds (`-RunProfileName` param absent, `RunStepResults` empty). Resolved by using `csexport.exe <connector> out.xml /f:x` to dump errored connector-space objects with the export-error detail.
|
||
- **UOS controller login rate-limit (HTTP 429)** on back-to-back gw-control applies. Resolved with a short backoff before the second apply.
|
||
- **git push masked failures** — piping `git push` through `tail` returned tail's exit code, hiding non-fast-forward rejections when origin diverged (Howard's concurrent commits). Resolved by rebasing onto origin and re-pushing; verified ahead/behind 0/0 each time.
|
||
- **Mailprotector dead-end for starrpass.com** — `find-user` 404 and no `starrpass.com` domain; the MP "Starr Pass" account (16170) only covers devconllc.com. User clarified starrpass.com is direct-to-MS; pivoted to remediation-tool/EOP. Saved as a reference memory + correction.
|
||
- **`get-token.sh` vault_path** — reads `~/.claude/identity.json` (lacks the field here); fixed by exporting `VAULT_ROOT_ENV=$(jq -r .vault_path .claude/identity.json)` (logged as friction earlier in session).
|
||
|
||
## Configuration Changes
|
||
|
||
- **Created** `.claude/skills/unifi-wifi/scripts/gw-control.sh` — gateway port-forward + WAN firewall + block-ips actions (commit eb87710b).
|
||
- **Edited** `.claude/skills/unifi-wifi/SKILL.md` + `references/ROADMAP.md` — documented gw-control; added pfSense compatibility-layer roadmap § E (eb87710b, 69987190). (Merged cleanly with Howard's concurrent monitor-run.sh additions.)
|
||
- **Edited** `.claude/scripts/onboarding-diagnostic.ps1` — Server-SKU EOL map (branch on `$caption -match 'Server'`) + disk-error provider filter (drop Kernel-Boot 153 false positive) (commit 313fd0ac).
|
||
- **Edited** `.claude/commands/syncro.md` — invoice `note` policy: `set_invoice_note` helper (upsell / hours-remaining / low-block + Winter tag), wired into billing Step 3, plus recurring-invoice sweep (commits 864f4d0a, a32dfc33).
|
||
- **Created** memories: `reference_aadconnect_keycredlink_writeback.md`, `reference_starrpass_mail_routing.md` (+ MEMORY.md index lines).
|
||
- **Created** `clients/grabb-durando/onboarding-baselines/GND-SERVER-20260616T151038.{json,md}` (RED baseline, commit a33bc423).
|
||
- **RMM client created:** "Russo, Steve" + sites Main / Shannon.
|
||
- **Vault adds:** Russo site keys, gd.local domain Administrator (see Credentials).
|
||
- **Live changes:** Grabb USG port-forward `VPN` (80,443,1723→80,443) + WAN_IN `GRE` disabled; GND-SERVER `tempadmin` disabled, `Administrator` pw rotated, scheduled task `ACG-MaintenanceReboot`; Glaztech CAS account UPN → CAS2@glaztech.com; AdminSDHolder ACEs on rrs-law.com and glaztech.com; Syncro contact 277990 email updated.
|
||
|
||
## Credentials & Secrets
|
||
|
||
- **gd.local Domain Administrator** — password rotated 2026-06-16 (24-char, crypto-random), vaulted at `clients/grabb-durando/gd-local-domain-admin.sops.yaml` (username `Administrator`, domain `gd.local`). NOTE: the rotation transited the GuruRMM command log (base64, admin-only) — rotate again if that log is a concern. Was unchanged since 2019.
|
||
- **Russo (Russo Law) GuruRMM site enrollment keys** — vaulted `clients/russo/gururmm-site-main.sops.yaml` (site_code SWIFT-RIVER-6063) and `clients/russo/gururmm-site-shannon.sops.yaml` (COPPER-HAWK-2295). client_id b83b0c29-81e6-4ce4-98ec-74639998d2ce.
|
||
- No other new secrets. Connector accounts referenced (not secrets): rrs-law.com `MSOL_080e1bfda418`, glaztech.com `MSOL_eb9a1b4a06ca`.
|
||
|
||
## Infrastructure & Servers
|
||
|
||
- **RUSSO-SRV** — Russo Law (rrs-law.com), Windows Server 2016 **DC**, GuruRMM agent `d10785ad-aeae-436d-96e0-5c9df5a9c754`. Entra Connect host.
|
||
- **GTI-INV-DC** — Glaztech (glaztech.com), Server 2016 **DC**, agent `0337e973-8d84-440e-9eb7-0c8af84efe70`. Entra Connect host (NOT GTI-INV-DC1).
|
||
- **GND-SERVER** — Grabb & Durando (gd.local), Server 2019 **DC** / file / RRAS, `192.168.242.200`, agent `cd086074-6766-46b5-93ad-382df97b1f54`. Whitebox (MSI MS-7B87, Ryzen 5 2600, 16 GB). Only RMM agent in Grabb's Main Office site.
|
||
- **Grabb USG-3P** — UGW3, WAN `174.76.185.203`, LAN `192.168.242.0/24`; UOS site id `6080d597f91fdd010f7c7155` ("Grabb and Durando"); also a US24 switch (`192.168.242.227`).
|
||
- **UOS controller** — `172.16.3.29:11443`; RW admin vaulted at `infrastructure/uos-server-network-api-rw`.
|
||
- **starrpass.com** — M365 tenant `222450dd-141f-435f-87b8-cec719aac99e` (direct-to-MS/EOP). Mailprotector "Starr Pass" account `16170` covers only `devconllc.com` (domain id `27629`).
|
||
- Tenant IDs referenced: rrs-law.com `bef1b190-f78f-4b1c-aa4b-fab186a30702`; glaztech.com `82931e3c-de7a-4f74-87f7-fe714be1f160`.
|
||
|
||
## Commands & Outputs
|
||
|
||
- AAD Connect export-error dump: `& "$env:ProgramFiles\Microsoft Azure AD Sync\Bin\csexport.exe" "<AD-connector>" out.xml /f:x` → `//cs-object/export-errordetail` gives `error-type=permission-issue`, `cd-error/error-code=8344`.
|
||
- AdminSDHolder fix: `dsacls "CN=AdminSDHolder,CN=System,DC=<dom>,DC=com" /G "<NETBIOS>\MSOL_xxx:WP;<attr>"` then rootDSE `RunProtectAdminGroupsTask=1` + `Start-ADSyncSyncCycle -PolicyType Delta`.
|
||
- gw-control: `bash .claude/skills/unifi-wifi/scripts/gw-control.sh "Grabb and Durando" pf-set-ports VPN 80,443 --apply` and `... fw-disable GRE --apply`. Verify with `pf-list` / `fw-list` (Mongo, no cred).
|
||
- Grabb VPN brute-force: 975×evt 20271 (failed auth) + 915×evt 20255 (PPP) in 14d; top IPs 91.92.240.84 (530), 178.16.55.208 (351); 0 successful (20274/20250/20272).
|
||
- EOP: `Get-MessageTraceV2` / `Get-QuarantineMessage` / `Get-MessageTraceDetailV2` via `POST https://outlook.office365.com/adminapi/beta/<tid>/InvokeCommand`. cansley fail reason: `550 5.1.10 RESOLVER.ADR.RecipientNotFound`.
|
||
- Syncro invoice note (= Invoice Message): `PUT /invoices/{id}` body `{"note":"..."}` → `{"invoice":{...}}`. Validated on test invoice #67741 (cust 15353550). Block product 46303 "Prepaid Hours - Block" @ $130/hr.
|
||
|
||
## Pending / Incomplete Tasks
|
||
|
||
- **GND-SERVER scheduled reboot** fires tonight 21:00 MST (`ACG-MaintenanceReboot`) — clears the print-driver pending reboot + 1 pending update. Cancel with `shutdown /a` if needed.
|
||
- **GND-SERVER remaining health items (1–2):** host firewall is OFF on all profiles (re-enable carefully on a DC); confirm AV/Defender coverage (no Bitdefender present); confirm Datto backup jobs actually succeed (portal); consider deleting `tempadmin` after a soak; BitLocker/no-encryption question.
|
||
- **Grabb VPN long-term:** decide whether to fully retire the unused RRAS VPN or stand up secure SSTP/IKEv2 (offered to client on #32427).
|
||
- **Recurring invoice-note sweep** is on-demand (Syncro auto-generates recurring invoices) — optionally schedule it as a cloud agent the day after each recurring run.
|
||
- **pfSense compatibility layer** (unifi-wifi ROADMAP § E) — design captured; build pending (validate on office + Cascades).
|
||
- **Wiki:** run `/wiki-compile` separately for grabb-durando, glaztech, russo-law if desired.
|
||
|
||
## Reference Information
|
||
|
||
- Commits: `eb87710b` (gw-control), `69987190` (pfSense roadmap), `313fd0ac` (probe fixes), `a33bc423` (GND baseline), `864f4d0a` + `a32dfc33` (syncro invoice-note policy), plus memory commits.
|
||
- Tickets: #32302 (Russo AAD Connect, Resolved), #32426 (Glaztech AAD Connect, Resolved), #32427 (Grabb emergency PPTP, Resolved), #32424 (Starr Pass block-labor comment + contact email update).
|
||
- Winter Discord id `624666486362996755` (low-block alert mention, #bot-alerts).
|
||
- Coord: lock `ce219518` (released), messages to Howard re gw-control (82a4acb4, 86ba01e4).
|
||
- Syncro: customers Russo Law 23331699, Glaztech 143932, Grabb 14232794, Starr Pass Realty 153298 (contact Chris Ansley id 277990); internal test customer 15353550.
|