diff --git a/.claude/memory/MEMORY.md b/.claude/memory/MEMORY.md index 33d1474..cfeaa3b 100644 --- a/.claude/memory/MEMORY.md +++ b/.claude/memory/MEMORY.md @@ -28,6 +28,7 @@ - [Trebesch DESKTOP-QNP3ON5 shell replacement](reference_trebesch_qnp3on5.md) — AT Trebesch box runs an Explorer shell replacement; explorer.exe owner check returns blank — use Win32_ComputerSystem.UserName. GuruRMM SWIFT-LION-2892. - [reference_backblaze_storage_rate](reference_backblaze_storage_rate.md) -- ACG's Backblaze B2 storage cost rate ($0.00695/GB) for the GuruRMM mspbackups storage-cost calculation - [Unraid VM no-IP causes](unraid-windows-vm-virtio-no-ip.md) — PRIMARY (general "new VMs stopped getting IPs lately"): Docker sets bridge-nf-call-iptables=1, so br0 VM DHCP OFFERs hit DOCKER-FORWARD (no br0 ACCEPT) and get dropped; new VMs can't complete DORA (existing renew via ESTABLISHED). Fix `=0` runtime (needs persistent post-Docker hook; not yet persisted on Jupiter). SECONDARY (Windows VM): virtio-net has no in-box driver -> use e1000 or virtio-win. Diagnose: tcpdump DHCP on pfSense; /sys vnetN rx_packets. +- [Starr Pass mail routing](reference_starrpass_mail_routing.md) — starrpass.com is DIRECT to MS (EOP/Defender, tenant 222450dd…); only devconllc.com is on Mailprotector (MP acct 16170). Check @starrpass.com quarantine/rejects via remediation-tool, not Mailprotector. - [AAD Connect msDS-KeyCredentialLink writeback](reference_aadconnect_keycredlink_writeback.md) — "completed-export-errors" + 8344 INSUFF_ACCESS_RIGHTS on a protected admin account = WHfB key writeback blocked by AdminSDHolder. Diagnose with csexport /f:x; fix with dsacls WP;msDS-KeyCredentialLink on AdminSDHolder + SDProp. - [reference_sqlx_migrations_immutable](reference_sqlx_migrations_immutable.md) -- NEVER edit an already-applied sqlx migration file — even a comment. sqlx::migrate! checksums each file at compile time and validates against _sqlx_migrations at startup; a changed checksum crash-loops the server with "migration N was previously applied but has been modified". Code review MUST flag any edit to an applied migration. diff --git a/.claude/memory/reference_starrpass_mail_routing.md b/.claude/memory/reference_starrpass_mail_routing.md new file mode 100644 index 0000000..b8aa7df --- /dev/null +++ b/.claude/memory/reference_starrpass_mail_routing.md @@ -0,0 +1,23 @@ +--- +name: reference_starrpass_mail_routing +description: Starr Pass mail routing — starrpass.com is DIRECT to Microsoft (EOP/Defender), NOT Mailprotector; only devconllc.com is on Mailprotector. Check quarantine/rejects accordingly. +metadata: + type: reference +--- + +**Starr Pass** email routing (don't reach for Mailprotector first): + +- **starrpass.com** delivers **direct to Microsoft** — M365 tenant `starrpass.com` + (tenant id `222450dd-141f-435f-87b8-cec719aac99e`). Quarantined / rejected / held / message-trace + questions for an @starrpass.com address = **EOP / Defender**, via the `remediation-tool` + (`investigator-exo` → EXO REST InvokeCommand: `Get-QuarantineMessage`, `Get-MessageTraceV2`, + `Get-MessageTraceDetailV2`). NOT on the Mailprotector/CloudFilter platform. +- The Mailprotector **"Starr Pass" account (id 16170)** covers ONLY the domain **devconllc.com** + (Devcon LLC, their management company) — domain id `27629`. So a Mailprotector `find-user + cansley@starrpass.com` 404s and `starrpass.com` is absent from the MP domain list — that's expected, + not a fault. + +Practical: for any @starrpass.com mail-flow ask, go straight to the remediation-tool/EOP. Use +Mailprotector only for devconllc.com. (Confirmed by Mike 2026-06-16 while checking quarantined/rejected +mail for cansley@starrpass.com — the "rejected" mail was `550 5.1.10 RecipientNotFound` from before the +mailbox was provisioned, now delivering.) Related: [[reference_resource_map]]. diff --git a/errorlog.md b/errorlog.md index 283363b..ec13de7 100644 --- a/errorlog.md +++ b/errorlog.md @@ -17,6 +17,10 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure · +2026-06-16 | GURU-5070 | mailprotector/starrpass | [correction] assumed starrpass.com was on Mailprotector; correct: starrpass.com is direct-to-MS (EOP/Defender) - Starr Pass MP account 16170 covers ONLY devconllc.com. Check @starrpass.com mail via remediation-tool/EOP + +2026-06-16 | GURU-5070 | mailprotector | HTTP 404 POST https://emailservice.io/api/v1/users/find_by_address: "Not found" [ctx: cmd=find-user] + 2026-06-16 | Howard-Home | bash/curl.exe-on-windows | [friction] PowerShell-invoked curl.exe strips embedded double-quotes from --data-urlencode args (CommandLineToArgvW), silently mangling POST bodies; pfSense PHP became 'echo PHPRUNS-OK' -> 'Undefined constant'. Fix: write payloads with single-quotes only, build $ via [char]36, keep one line. [ctx: ref=pfsense diag_command.php php-exec; cost=4 wasted RMM round-trips] 2026-06-16 | GURU-5070 | remediation-tool/get-token | [friction] get-token.sh reads vault_path from ~/.claude/identity.json (home), which lacks the field on this machine; repo identity.json (.claude/identity.json) has it. Fix: export VAULT_ROOT_ENV=$(jq -r .vault_path .claude/identity.json) before calling get-token [ctx: ref=remediation-tool;machine=GURU-5070] diff --git a/session-logs/2026-06/2026-06-16-mike-adsync-grabb-vpn-syncro-automation.md b/session-logs/2026-06/2026-06-16-mike-adsync-grabb-vpn-syncro-automation.md new file mode 100644 index 0000000..99c2bb8 --- /dev/null +++ b/session-logs/2026-06/2026-06-16-mike-adsync-grabb-vpn-syncro-automation.md @@ -0,0 +1,89 @@ +## User +- **User:** Mike Swanson (mike) +- **Machine:** GURU-5070 +- **Role:** admin + +## Session Summary + +A long multi-client MSP session spanning Entra Connect (AAD Connect) remediation, an internet-facing VPN brute-force takedown, new UniFi gateway tooling, a server health/security hardening pass, an EOP mail investigation, and a Syncro invoice-automation build. + +Started by provisioning a new GuruRMM client "Russo, Steve" (Russo Law Firm) with two sites (Main, Shannon) and vaulting both one-time enrollment keys. The user then reported "Russo is having an ad sync issue." Diagnosis on RUSSO-SRV (rrs-law.com DC) found Entra Connect healthy except one persistent AD-connector export error: writeback of `msDS-KeyCredentialLink` (Windows Hello key) to `guru@rrs-law.com` denied with LDAP 8344, failing since 2025-05-07 (17.7k retries). Root cause: the account is AdminSDHolder-protected (Domain/Enterprise/Schema Admins), so SDProp strips the connector account's write permission. Fixed by delegating `WP;msDS-KeyCredentialLink` on AdminSDHolder via dsacls + forcing SDProp + a delta sync; verified zero export errors. Ticketed/billed #32302 (1 hr warranty, public+emailed resolution). + +Repeated the same diagnosis on Glaztech (GTI-INV-DC, glaztech.com DC) — identical fault on a different attribute: `msExchSafeSendersHash` (Exchange hybrid writeback) denied on `seastman` since 2025-08-28. Applied the same AdminSDHolder fix. Cloud-side also surfaced a separate UPN PropertyConflict: `CAS@glaztech.com` was both the `CAS` user's UPN and an alias on Alex Miramontez, parking the synced CAS object as `CAS1944@onmicrosoft`. Resolved by changing the CAS account's on-prem UPN to its own primary (`CAS2@glaztech.com`) — conflict cleared, no mail-routing change. Ticketed/billed #32426 (1 hr remote, prepaid). + +Grabb & Durando: investigating GND-SERVER's "needs a reboot for VPN to work" symptom revealed an internet-facing **PPTP RRAS VPN under sustained brute-force** (~1,900 failed attempts/14 days, 0 legit logins). The edge is a UniFi USG-3P (WAN 174.76.185.203) port-forwarding 80/443/1723 from `src=any` to GND-SERVER (192.168.242.200). To remediate at the gateway, **built a new unifi-wifi skill script `gw-control.sh`** (port-forward + WAN firewall + block-ips actions, gated/DRY-RUN/rollback, same RW-REST pattern as apply-wlan), after pulling Howard's 15 newer unifi commits and coordinating via coord lock+messages. Applied Option A (`pf-set-ports VPN 80,443` to drop 1723 + `fw-disable GRE`), then restarted RRAS to free the wedged ports. Added a pfSense gateway "compatibility layer" roadmap item (§ E). Emergency-ticketed #32427 (0.5 h ×1.5 prepaid). + +Ran a full onboarding diagnostic on GND-SERVER (graded RED) and found two probe false-positives (Server-2019 flagged EOL via the Win10-1809 build collision; Kernel-Boot event 153 "VBS disabled" miscounted as a disk error) — fixed both in `onboarding-diagnostic.ps1`. Hardened the box: disabled the dormant domain admin `tempadmin` (unused since 2022), rotated the domain `Administrator` password (unchanged since 2019) and vaulted it, and scheduled an off-hours maintenance reboot. Confirmed no workstations are enrolled in Grabb's RMM site. Then checked EOP for `cansley@starrpass.com` (user corrected that starrpass.com is direct-to-Microsoft, not Mailprotector): 0 quarantined, 2 old Apple "rejections" that were `550 5.1.10 RecipientNotFound` from before the mailbox existed. Finally, built a Syncro invoice-note automation: the on-screen "Invoice Message" is the invoice `note` field, now auto-populated with a block-rate upsell hint (non-block customers), an hours-remaining count (block customers), and a low-block (<4 hr) renewal line that tags Winter in #bot-alerts; plus a recurring-invoice sweep. + +## Key Decisions + +- **Don't disable the domain built-in Administrator on a DC** — it's the break-glass account; rotated its stale (2019) password and vaulted instead, left enabled. Disabled the genuinely dormant `tempadmin` (reversible). +- **AdminSDHolder grant, not OU/object grant** — protected accounts have inheritance stripped by SDProp, so the permission must live on AdminSDHolder to persist; `dsacls /G` is additive (one attribute-scoped ACE). +- **CAS UPN conflict fixed by renaming the CAS account's UPN** (to its own primary CAS2@glaztech.com), not by removing Alex's alias — zero mail-routing change, since CAS@ already routes to Alex. +- **Grabb VPN: close PPTP at the USG, keep 443/80** — dropped only 1723 + GRE (PPTP needs both), preserving SSTP/web; 0 legit logins in 14 days made it low-risk. +- **Built gw-control.sh rather than one-off API calls** — the firewall/port-forward gap was the remaining unifi ROADMAP item; coordinated with Howard (who was actively shipping) via lock + messages to avoid collision. +- **`< 4 hrs` absolute threshold for low-block** (user's simplification) instead of "25% of full block," because the API's `prepay_hours` is only the current balance — full block size isn't exposed. +- **Recurring invoice notes via a post-run sweep** — Syncro generates recurring invoices server-side, so a dynamic per-remaining-hours note requires an idempotent sweep, not a static schedule note. +- **Fixed probe false-positives at the source** rather than just flagging — both bugs would mis-grade every Windows Server in the fleet. + +## Problems Encountered + +- **AAD Connect run-step detail not exposed by `Get-ADSyncRunProfileResult`** on the older builds (`-RunProfileName` param absent, `RunStepResults` empty). Resolved by using `csexport.exe out.xml /f:x` to dump errored connector-space objects with the export-error detail. +- **UOS controller login rate-limit (HTTP 429)** on back-to-back gw-control applies. Resolved with a short backoff before the second apply. +- **git push masked failures** — piping `git push` through `tail` returned tail's exit code, hiding non-fast-forward rejections when origin diverged (Howard's concurrent commits). Resolved by rebasing onto origin and re-pushing; verified ahead/behind 0/0 each time. +- **Mailprotector dead-end for starrpass.com** — `find-user` 404 and no `starrpass.com` domain; the MP "Starr Pass" account (16170) only covers devconllc.com. User clarified starrpass.com is direct-to-MS; pivoted to remediation-tool/EOP. Saved as a reference memory + correction. +- **`get-token.sh` vault_path** — reads `~/.claude/identity.json` (lacks the field here); fixed by exporting `VAULT_ROOT_ENV=$(jq -r .vault_path .claude/identity.json)` (logged as friction earlier in session). + +## Configuration Changes + +- **Created** `.claude/skills/unifi-wifi/scripts/gw-control.sh` — gateway port-forward + WAN firewall + block-ips actions (commit eb87710b). +- **Edited** `.claude/skills/unifi-wifi/SKILL.md` + `references/ROADMAP.md` — documented gw-control; added pfSense compatibility-layer roadmap § E (eb87710b, 69987190). (Merged cleanly with Howard's concurrent monitor-run.sh additions.) +- **Edited** `.claude/scripts/onboarding-diagnostic.ps1` — Server-SKU EOL map (branch on `$caption -match 'Server'`) + disk-error provider filter (drop Kernel-Boot 153 false positive) (commit 313fd0ac). +- **Edited** `.claude/commands/syncro.md` — invoice `note` policy: `set_invoice_note` helper (upsell / hours-remaining / low-block + Winter tag), wired into billing Step 3, plus recurring-invoice sweep (commits 864f4d0a, a32dfc33). +- **Created** memories: `reference_aadconnect_keycredlink_writeback.md`, `reference_starrpass_mail_routing.md` (+ MEMORY.md index lines). +- **Created** `clients/grabb-durando/onboarding-baselines/GND-SERVER-20260616T151038.{json,md}` (RED baseline, commit a33bc423). +- **RMM client created:** "Russo, Steve" + sites Main / Shannon. +- **Vault adds:** Russo site keys, gd.local domain Administrator (see Credentials). +- **Live changes:** Grabb USG port-forward `VPN` (80,443,1723→80,443) + WAN_IN `GRE` disabled; GND-SERVER `tempadmin` disabled, `Administrator` pw rotated, scheduled task `ACG-MaintenanceReboot`; Glaztech CAS account UPN → CAS2@glaztech.com; AdminSDHolder ACEs on rrs-law.com and glaztech.com; Syncro contact 277990 email updated. + +## Credentials & Secrets + +- **gd.local Domain Administrator** — password rotated 2026-06-16 (24-char, crypto-random), vaulted at `clients/grabb-durando/gd-local-domain-admin.sops.yaml` (username `Administrator`, domain `gd.local`). NOTE: the rotation transited the GuruRMM command log (base64, admin-only) — rotate again if that log is a concern. Was unchanged since 2019. +- **Russo (Russo Law) GuruRMM site enrollment keys** — vaulted `clients/russo/gururmm-site-main.sops.yaml` (site_code SWIFT-RIVER-6063) and `clients/russo/gururmm-site-shannon.sops.yaml` (COPPER-HAWK-2295). client_id b83b0c29-81e6-4ce4-98ec-74639998d2ce. +- No other new secrets. Connector accounts referenced (not secrets): rrs-law.com `MSOL_080e1bfda418`, glaztech.com `MSOL_eb9a1b4a06ca`. + +## Infrastructure & Servers + +- **RUSSO-SRV** — Russo Law (rrs-law.com), Windows Server 2016 **DC**, GuruRMM agent `d10785ad-aeae-436d-96e0-5c9df5a9c754`. Entra Connect host. +- **GTI-INV-DC** — Glaztech (glaztech.com), Server 2016 **DC**, agent `0337e973-8d84-440e-9eb7-0c8af84efe70`. Entra Connect host (NOT GTI-INV-DC1). +- **GND-SERVER** — Grabb & Durando (gd.local), Server 2019 **DC** / file / RRAS, `192.168.242.200`, agent `cd086074-6766-46b5-93ad-382df97b1f54`. Whitebox (MSI MS-7B87, Ryzen 5 2600, 16 GB). Only RMM agent in Grabb's Main Office site. +- **Grabb USG-3P** — UGW3, WAN `174.76.185.203`, LAN `192.168.242.0/24`; UOS site id `6080d597f91fdd010f7c7155` ("Grabb and Durando"); also a US24 switch (`192.168.242.227`). +- **UOS controller** — `172.16.3.29:11443`; RW admin vaulted at `infrastructure/uos-server-network-api-rw`. +- **starrpass.com** — M365 tenant `222450dd-141f-435f-87b8-cec719aac99e` (direct-to-MS/EOP). Mailprotector "Starr Pass" account `16170` covers only `devconllc.com` (domain id `27629`). +- Tenant IDs referenced: rrs-law.com `bef1b190-f78f-4b1c-aa4b-fab186a30702`; glaztech.com `82931e3c-de7a-4f74-87f7-fe714be1f160`. + +## Commands & Outputs + +- AAD Connect export-error dump: `& "$env:ProgramFiles\Microsoft Azure AD Sync\Bin\csexport.exe" "" out.xml /f:x` → `//cs-object/export-errordetail` gives `error-type=permission-issue`, `cd-error/error-code=8344`. +- AdminSDHolder fix: `dsacls "CN=AdminSDHolder,CN=System,DC=,DC=com" /G "\MSOL_xxx:WP;"` then rootDSE `RunProtectAdminGroupsTask=1` + `Start-ADSyncSyncCycle -PolicyType Delta`. +- gw-control: `bash .claude/skills/unifi-wifi/scripts/gw-control.sh "Grabb and Durando" pf-set-ports VPN 80,443 --apply` and `... fw-disable GRE --apply`. Verify with `pf-list` / `fw-list` (Mongo, no cred). +- Grabb VPN brute-force: 975×evt 20271 (failed auth) + 915×evt 20255 (PPP) in 14d; top IPs 91.92.240.84 (530), 178.16.55.208 (351); 0 successful (20274/20250/20272). +- EOP: `Get-MessageTraceV2` / `Get-QuarantineMessage` / `Get-MessageTraceDetailV2` via `POST https://outlook.office365.com/adminapi/beta//InvokeCommand`. cansley fail reason: `550 5.1.10 RESOLVER.ADR.RecipientNotFound`. +- Syncro invoice note (= Invoice Message): `PUT /invoices/{id}` body `{"note":"..."}` → `{"invoice":{...}}`. Validated on test invoice #67741 (cust 15353550). Block product 46303 "Prepaid Hours - Block" @ $130/hr. + +## Pending / Incomplete Tasks + +- **GND-SERVER scheduled reboot** fires tonight 21:00 MST (`ACG-MaintenanceReboot`) — clears the print-driver pending reboot + 1 pending update. Cancel with `shutdown /a` if needed. +- **GND-SERVER remaining health items (1–2):** host firewall is OFF on all profiles (re-enable carefully on a DC); confirm AV/Defender coverage (no Bitdefender present); confirm Datto backup jobs actually succeed (portal); consider deleting `tempadmin` after a soak; BitLocker/no-encryption question. +- **Grabb VPN long-term:** decide whether to fully retire the unused RRAS VPN or stand up secure SSTP/IKEv2 (offered to client on #32427). +- **Recurring invoice-note sweep** is on-demand (Syncro auto-generates recurring invoices) — optionally schedule it as a cloud agent the day after each recurring run. +- **pfSense compatibility layer** (unifi-wifi ROADMAP § E) — design captured; build pending (validate on office + Cascades). +- **Wiki:** run `/wiki-compile` separately for grabb-durando, glaztech, russo-law if desired. + +## Reference Information + +- Commits: `eb87710b` (gw-control), `69987190` (pfSense roadmap), `313fd0ac` (probe fixes), `a33bc423` (GND baseline), `864f4d0a` + `a32dfc33` (syncro invoice-note policy), plus memory commits. +- Tickets: #32302 (Russo AAD Connect, Resolved), #32426 (Glaztech AAD Connect, Resolved), #32427 (Grabb emergency PPTP, Resolved), #32424 (Starr Pass block-labor comment + contact email update). +- Winter Discord id `624666486362996755` (low-block alert mention, #bot-alerts). +- Coord: lock `ce219518` (released), messages to Howard re gw-control (82a4acb4, 86ba01e4). +- Syncro: customers Russo Law 23331699, Glaztech 143932, Grabb 14232794, Starr Pass Realty 153298 (contact Chris Ansley id 277990); internal test customer 15353550.