claudetools/session-logs/2026-06/2026-06-16-mike-adsync-grabb-vpn-syncro-automation.md

User

• User: Mike Swanson (mike)
• Machine: GURU-5070
• Role: admin

Session Summary

A long multi-client MSP session spanning Entra Connect (AAD Connect) remediation, an internet-facing VPN brute-force takedown, new UniFi gateway tooling, a server health/security hardening pass, an EOP mail investigation, and a Syncro invoice-automation build.

Started by provisioning a new GuruRMM client "Russo, Steve" (Russo Law Firm) with two sites (Main, Shannon) and vaulting both one-time enrollment keys. The user then reported "Russo is having an ad sync issue." Diagnosis on RUSSO-SRV (rrs-law.com DC) found Entra Connect healthy except one persistent AD-connector export error: writeback of msDS-KeyCredentialLink (Windows Hello key) to guru@rrs-law.com denied with LDAP 8344, failing since 2025-05-07 (17.7k retries). Root cause: the account is AdminSDHolder-protected (Domain/Enterprise/Schema Admins), so SDProp strips the connector account's write permission. Fixed by delegating WP;msDS-KeyCredentialLink on AdminSDHolder via dsacls + forcing SDProp + a delta sync; verified zero export errors. Ticketed/billed #32302 (1 hr warranty, public+emailed resolution).

Repeated the same diagnosis on Glaztech (GTI-INV-DC, glaztech.com DC) — identical fault on a different attribute: msExchSafeSendersHash (Exchange hybrid writeback) denied on seastman since 2025-08-28. Applied the same AdminSDHolder fix. Cloud-side also surfaced a separate UPN PropertyConflict: CAS@glaztech.com was both the CAS user's UPN and an alias on Alex Miramontez, parking the synced CAS object as CAS1944@onmicrosoft. Resolved by changing the CAS account's on-prem UPN to its own primary (CAS2@glaztech.com) — conflict cleared, no mail-routing change. Ticketed/billed #32426 (1 hr remote, prepaid).

Grabb & Durando: investigating GND-SERVER's "needs a reboot for VPN to work" symptom revealed an internet-facing PPTP RRAS VPN under sustained brute-force (~1,900 failed attempts/14 days, 0 legit logins). The edge is a UniFi USG-3P (WAN 174.76.185.203) port-forwarding 80/443/1723 from src=any to GND-SERVER (192.168.242.200). To remediate at the gateway, built a new unifi-wifi skill script gw-control.sh (port-forward + WAN firewall + block-ips actions, gated/DRY-RUN/rollback, same RW-REST pattern as apply-wlan), after pulling Howard's 15 newer unifi commits and coordinating via coord lock+messages. Applied Option A (pf-set-ports VPN 80,443 to drop 1723 + fw-disable GRE), then restarted RRAS to free the wedged ports. Added a pfSense gateway "compatibility layer" roadmap item (§ E). Emergency-ticketed #32427 (0.5 h ×1.5 prepaid).

Ran a full onboarding diagnostic on GND-SERVER (graded RED) and found two probe false-positives (Server-2019 flagged EOL via the Win10-1809 build collision; Kernel-Boot event 153 "VBS disabled" miscounted as a disk error) — fixed both in onboarding-diagnostic.ps1. Hardened the box: disabled the dormant domain admin tempadmin (unused since 2022), rotated the domain Administrator password (unchanged since 2019) and vaulted it, and scheduled an off-hours maintenance reboot. Confirmed no workstations are enrolled in Grabb's RMM site. Then checked EOP for cansley@starrpass.com (user corrected that starrpass.com is direct-to-Microsoft, not Mailprotector): 0 quarantined, 2 old Apple "rejections" that were 550 5.1.10 RecipientNotFound from before the mailbox existed. Finally, built a Syncro invoice-note automation: the on-screen "Invoice Message" is the invoice note field, now auto-populated with a block-rate upsell hint (non-block customers), an hours-remaining count (block customers), and a low-block (<4 hr) renewal line that tags Winter in #bot-alerts; plus a recurring-invoice sweep.

Key Decisions

• Don't disable the domain built-in Administrator on a DC — it's the break-glass account; rotated its stale (2019) password and vaulted instead, left enabled. Disabled the genuinely dormant tempadmin (reversible).
• AdminSDHolder grant, not OU/object grant — protected accounts have inheritance stripped by SDProp, so the permission must live on AdminSDHolder to persist; dsacls /G is additive (one attribute-scoped ACE).
• CAS UPN conflict fixed by renaming the CAS account's UPN (to its own primary CAS2@glaztech.com), not by removing Alex's alias — zero mail-routing change, since CAS@ already routes to Alex.
• Grabb VPN: close PPTP at the USG, keep 443/80 — dropped only 1723 + GRE (PPTP needs both), preserving SSTP/web; 0 legit logins in 14 days made it low-risk.
• Built gw-control.sh rather than one-off API calls — the firewall/port-forward gap was the remaining unifi ROADMAP item; coordinated with Howard (who was actively shipping) via lock + messages to avoid collision.
• < 4 hrs absolute threshold for low-block (user's simplification) instead of "25% of full block," because the API's prepay_hours is only the current balance — full block size isn't exposed.
• Recurring invoice notes via a post-run sweep — Syncro generates recurring invoices server-side, so a dynamic per-remaining-hours note requires an idempotent sweep, not a static schedule note.
• Fixed probe false-positives at the source rather than just flagging — both bugs would mis-grade every Windows Server in the fleet.

Problems Encountered

• AAD Connect run-step detail not exposed by Get-ADSyncRunProfileResult on the older builds (-RunProfileName param absent, RunStepResults empty). Resolved by using csexport.exe <connector> out.xml /f:x to dump errored connector-space objects with the export-error detail.
• UOS controller login rate-limit (HTTP 429) on back-to-back gw-control applies. Resolved with a short backoff before the second apply.
• git push masked failures — piping git push through tail returned tail's exit code, hiding non-fast-forward rejections when origin diverged (Howard's concurrent commits). Resolved by rebasing onto origin and re-pushing; verified ahead/behind 0/0 each time.
• Mailprotector dead-end for starrpass.com — find-user 404 and no starrpass.com domain; the MP "Starr Pass" account (16170) only covers devconllc.com. User clarified starrpass.com is direct-to-MS; pivoted to remediation-tool/EOP. Saved as a reference memory + correction.
• get-token.sh vault_path — reads ~/.claude/identity.json (lacks the field here); fixed by exporting VAULT_ROOT_ENV=$(jq -r .vault_path .claude/identity.json) (logged as friction earlier in session).

Configuration Changes

• Created .claude/skills/unifi-wifi/scripts/gw-control.sh — gateway port-forward + WAN firewall + block-ips actions (commit eb87710b).
• Edited .claude/skills/unifi-wifi/SKILL.md + references/ROADMAP.md — documented gw-control; added pfSense compatibility-layer roadmap § E (eb87710b, 69987190). (Merged cleanly with Howard's concurrent monitor-run.sh additions.)
• Edited .claude/scripts/onboarding-diagnostic.ps1 — Server-SKU EOL map (branch on $caption -match 'Server') + disk-error provider filter (drop Kernel-Boot 153 false positive) (commit 313fd0ac).
• Edited .claude/commands/syncro.md — invoice note policy: set_invoice_note helper (upsell / hours-remaining / low-block + Winter tag), wired into billing Step 3, plus recurring-invoice sweep (commits 864f4d0a, a32dfc33).
• Created memories: reference_aadconnect_keycredlink_writeback.md, reference_starrpass_mail_routing.md (+ MEMORY.md index lines).
• Created clients/grabb-durando/onboarding-baselines/GND-SERVER-20260616T151038.{json,md} (RED baseline, commit a33bc423).
• RMM client created: "Russo, Steve" + sites Main / Shannon.
• Vault adds: Russo site keys, gd.local domain Administrator (see Credentials).
• Live changes: Grabb USG port-forward VPN (80,443,1723→80,443) + WAN_IN GRE disabled; GND-SERVER tempadmin disabled, Administrator pw rotated, scheduled task ACG-MaintenanceReboot; Glaztech CAS account UPN → CAS2@glaztech.com; AdminSDHolder ACEs on rrs-law.com and glaztech.com; Syncro contact 277990 email updated.

Credentials & Secrets

• gd.local Domain Administrator — password rotated 2026-06-16 (24-char, crypto-random), vaulted at clients/grabb-durando/gd-local-domain-admin.sops.yaml (username Administrator, domain gd.local). NOTE: the rotation transited the GuruRMM command log (base64, admin-only) — rotate again if that log is a concern. Was unchanged since 2019.
• Russo (Russo Law) GuruRMM site enrollment keys — vaulted clients/russo/gururmm-site-main.sops.yaml (site_code SWIFT-RIVER-6063) and clients/russo/gururmm-site-shannon.sops.yaml (COPPER-HAWK-2295). client_id b83b0c29-81e6-4ce4-98ec-74639998d2ce.
• No other new secrets. Connector accounts referenced (not secrets): rrs-law.com MSOL_080e1bfda418, glaztech.com MSOL_eb9a1b4a06ca.

Infrastructure & Servers

• RUSSO-SRV — Russo Law (rrs-law.com), Windows Server 2016 DC, GuruRMM agent d10785ad-aeae-436d-96e0-5c9df5a9c754. Entra Connect host.
• GTI-INV-DC — Glaztech (glaztech.com), Server 2016 DC, agent 0337e973-8d84-440e-9eb7-0c8af84efe70. Entra Connect host (NOT GTI-INV-DC1).
• GND-SERVER — Grabb & Durando (gd.local), Server 2019 DC / file / RRAS, 192.168.242.200, agent cd086074-6766-46b5-93ad-382df97b1f54. Whitebox (MSI MS-7B87, Ryzen 5 2600, 16 GB). Only RMM agent in Grabb's Main Office site.
• Grabb USG-3P — UGW3, WAN 174.76.185.203, LAN 192.168.242.0/24; UOS site id 6080d597f91fdd010f7c7155 ("Grabb and Durando"); also a US24 switch (192.168.242.227).
• UOS controller — 172.16.3.29:11443; RW admin vaulted at infrastructure/uos-server-network-api-rw.
• starrpass.com — M365 tenant 222450dd-141f-435f-87b8-cec719aac99e (direct-to-MS/EOP). Mailprotector "Starr Pass" account 16170 covers only devconllc.com (domain id 27629).
• Tenant IDs referenced: rrs-law.com bef1b190-f78f-4b1c-aa4b-fab186a30702; glaztech.com 82931e3c-de7a-4f74-87f7-fe714be1f160.

Commands & Outputs

• AAD Connect export-error dump: & "$env:ProgramFiles\Microsoft Azure AD Sync\Bin\csexport.exe" "<AD-connector>" out.xml /f:x → //cs-object/export-errordetail gives error-type=permission-issue, cd-error/error-code=8344.
• AdminSDHolder fix: dsacls "CN=AdminSDHolder,CN=System,DC=<dom>,DC=com" /G "<NETBIOS>\MSOL_xxx:WP;<attr>" then rootDSE RunProtectAdminGroupsTask=1 + Start-ADSyncSyncCycle -PolicyType Delta.
• gw-control: bash .claude/skills/unifi-wifi/scripts/gw-control.sh "Grabb and Durando" pf-set-ports VPN 80,443 --apply and ... fw-disable GRE --apply. Verify with pf-list / fw-list (Mongo, no cred).
• Grabb VPN brute-force: 975×evt 20271 (failed auth) + 915×evt 20255 (PPP) in 14d; top IPs 91.92.240.84 (530), 178.16.55.208 (351); 0 successful (20274/20250/20272).
• EOP: Get-MessageTraceV2 / Get-QuarantineMessage / Get-MessageTraceDetailV2 via POST https://outlook.office365.com/adminapi/beta/<tid>/InvokeCommand. cansley fail reason: 550 5.1.10 RESOLVER.ADR.RecipientNotFound.
• Syncro invoice note (= Invoice Message): PUT /invoices/{id} body {"note":"..."} → {"invoice":{...}}. Validated on test invoice #67741 (cust 15353550). Block product 46303 "Prepaid Hours - Block" @ $130/hr.

Pending / Incomplete Tasks

• GND-SERVER scheduled reboot fires tonight 21:00 MST (ACG-MaintenanceReboot) — clears the print-driver pending reboot + 1 pending update. Cancel with shutdown /a if needed.
• GND-SERVER remaining health items (1–2): host firewall is OFF on all profiles (re-enable carefully on a DC); confirm AV/Defender coverage (no Bitdefender present); confirm Datto backup jobs actually succeed (portal); consider deleting tempadmin after a soak; BitLocker/no-encryption question.
• Grabb VPN long-term: decide whether to fully retire the unused RRAS VPN or stand up secure SSTP/IKEv2 (offered to client on #32427).
• Recurring invoice-note sweep is on-demand (Syncro auto-generates recurring invoices) — optionally schedule it as a cloud agent the day after each recurring run.
• pfSense compatibility layer (unifi-wifi ROADMAP § E) — design captured; build pending (validate on office + Cascades).
• Wiki: run /wiki-compile separately for grabb-durando, glaztech, russo-law if desired.

Reference Information

• Commits: eb87710b (gw-control), 69987190 (pfSense roadmap), 313fd0ac (probe fixes), a33bc423 (GND baseline), 864f4d0a + a32dfc33 (syncro invoice-note policy), plus memory commits.
• Tickets: #32302 (Russo AAD Connect, Resolved), #32426 (Glaztech AAD Connect, Resolved), #32427 (Grabb emergency PPTP, Resolved), #32424 (Starr Pass block-labor comment + contact email update).
• Winter Discord id 624666486362996755 (low-block alert mention, #bot-alerts).
• Coord: lock ce219518 (released), messages to Howard re gw-control (82a4acb4, 86ba01e4).
• Syncro: customers Russo Law 23331699, Glaztech 143932, Grabb 14232794, Starr Pass Realty 153298 (contact Chris Ansley id 277990); internal test customer 15353550.