114 KiB
114 KiB
type, name, display_name, last_compiled, compiled_by, sources, backlinks
| type | name | display_name | last_compiled | compiled_by | sources | backlinks | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| client | cascades-tucson | Cascades of Tucson | 2026-06-30 | Howard-Home/claude-main |
|
|
Cascades of Tucson
Senior living / assisted living facility in Tucson, AZ. Single 6-floor building plus a MemCare (Memory Care) wing on floors 5-6. ACG took over from a previous MSP. Primary compliance driver is HIPAA. Active multi-phase migration project ongoing as of 2026-06-30.
Entra Access Architecture (canonical overview)
In one line: a HIPAA-driven, identity-based access-control system that splits staff into two security postures and enforces them with Microsoft Entra Conditional Access on top of hybrid identity (Entra Connect), with ALIS (clinical EHR) wired for SSO. Tickets: #109412123 (Entra setup), #110680053 (domain migration).
Foundation -- hybrid identity
- On-prem AD
cascades.localsynced to Entra/M365 via Entra Connect (PHS + Seamless SSO). UPN suffixcascadestucson.com, so a user's Windows login = email = M365/ALIS identity (one credential everywhere).
Two user buckets (the core design)
- Restricted -- caregivers + medtechs (group
SG-Caregivers,8b8d9222): sign in only on the Cascades network and only on approved devices (shared Galaxy phones + a set of caregiver laptops/desktops). No MFA (no personal devices) -- protected by location + device controls + 8h sign-in frequency instead. Effect: caregiver credentials are useless off-site or off an approved device -- the anti-hacker / bad-employee-from-home control. - Privileged -- admins / directors / managers / nurses (NOT in
SG-Caregivers): email + ALIS from anywhere, seamless onsite / 2FA offsite (Authenticator/PIN). Untouched by the caregiver lockdown.
Conditional Access enforcement (caregivers)
CSC - Block caregivers off Cascades network(e35614e1)CSC - Block caregivers on non-compliant device(ede985e2) -- being replaced by a device allow-list (CSC - Caregivers: allow-listed devices only,1b7fd025): phones (displayName -startsWith "CSC-") + tagged caregiver machines (extensionAttribute1 -eq "CSCCaregiverDevice", or explicit deviceId). Note: extensionAttribute changes lag >70 min into CA's filter cache -- deviceId matching is the lag-free lever for the small device set.CSC - Caregiver sign-in frequency 8h(7d491c7a)- Rollout is per-user via group membership (test group
SG-Caregivers-DeviceTestdb5849eccarries the full rule set for one-at-a-time validation; promote toSG-Caregivers+ disable compliance-block when validated).
Devices
- Phones: Samsung A15s in Intune Shared Device Mode (Android Enterprise, device-token enrolled) -- live.
- Laptops/desktops: caregiver shared machines (Laptop2, LAPTOP-DRQ5L558, LAPTOP-E0STJJE8, ASSISTNURSE-PC, NURSESTATION-PC) joined to Entra so CA recognizes them and they go on the allow-list (group
Cascades - Caregiver Devices02c6f698for policy targeting).
ALIS SSO
- Entra app registration -> OIDC SSO into ALIS; tenant-wide admin consent granted (2026-06-03). Per-user join key = ALIS staff Email must equal the Entra UPN. Caregivers SSO silently on phones (ALIS-native 2FA off); office users SSO with offsite MFA.
Caregiver phone SSO go-live (SUBSTANTIALLY DONE 2026-06-30 -- Entra/identity side complete)
The caregiver phone-SSO onboarding was executed 2026-06-30. To silently SSO into ALIS on a shared Samsung phone, each caregiver must be (1) in SG-Caregivers (bypasses the tenant-wide all-users-MFA CA policy, falls under the location+device posture), (2) M365-licensed (Business Premium, which also carries the Entra ID P1 the CA lockdown needs per-user), and (3) have ALIS staff Email = Entra UPN. Live AD state: OU=Caregivers holds 42 objects = 40 enabled real caregivers + pilot.test (test artifact) + n.castro (disabled). All 40 real caregivers had NEVER logged into the domain (bulk-created 2026-05-16/18) and were UNLICENSED before this session.
- DONE (Entra/identity): all 40 caregivers added to
SG-Caregivers(was 38; addedc.lassey+p.sandoval-beck), assigned Microsoft 365 Business Premium (usageLocation=USfirst, thenassignLicense), and given unique phone-typeable AD temp passwords with forced change at next logon (hybrid PHS -> the AD password is also the M365/phone sign-in). Temp passwords vaulted atclients/cascades-tucson/caregiver-temp-passwords-2026-06-30.sops.yaml(40 entries; retrieve withvault get, NOT get-field -- keys contain dots; delivered to Howard via Discord DM).SG-Caregiversis frontline caregivers ONLY -- Veronica Feller + Christine Nyanzunda (admin-adjacent) andpilot.test/n.castroare intentionally excluded (reverses the earlier 6/4 plan to add Feller/Nyanzunda). - REMAINING gate (Howard handling -- ALIS side): set each caregiver's ALIS staff
Email= Entra UPN so "Sign in with Microsoft" resolves. Of the 40 AD caregivers: 23 confirmed ALIS caregivers (just need Email=UPN), 5 in ALIS with blank job role (confirm caregiver + Email=UPN), 5 Med Techs (Email=UPN; Howard earlier said "ignore for the moment" -- revisit), 7 have NO ALIS staff record (must be created before SSO can work), and 3 ALIS caregivers have no AD account (Judith Palmer, Joey Ty, Alejandra Vallejo -- create AD accounts if they need phones). Also blanket-disable ALIS-native 2FA for the caregiver bucket as records are matched. NOTE: Zeke Huerta stayse.huerta@cascadestucson.com(do NOT "correct" to z.huerta) -- his ALIS Email must bee.huerta@. Build path:alisskillbuild-import-> upload .xls in ALIS UI (no staff-write API). - Prior crosscheck (2026-06-29): phone-only caregivers = NONE (all caregiver rows are
D+P; only the 3 Transportation drivers are phone-only and do not need ALIS). 7 caregiver-list people are present in ALIS only as Discharged records (Niel Castro, Kasey Flores, Bella Mendoza, Corey Tate, Gloria Williford, Mary Kariuki [DUP records 429856/429858], Maia Baker) -- decide reactivate-vs-recreate. Confirm Charity Sika (CSV) == Bariffa Sika (ALIS 309045).
Caregiver desktop/laptop management -- Hybrid Entra Join + GPO (the chosen path)
Because per-user Intune never provisioned tenant-wide (INTUNE_A = PendingInput; no Windows device ever Intune-enrolled -- MS case open), Windows caregiver devices are managed via Hybrid Entra Join + on-prem Group Policy instead. This needs no Intune. The CA access model is unchanged (hybrid join just gives the device an Entra object so the allow-list/deviceId still applies).
- Hybrid join proven on NURSESTATION-PC (2026-06-05): SCP written (
ConfigureSCP.ps1),OU=Caregiver Devices,OU=Staff PCs,OU=Workstationsadded to Entra Connect sync scope -> device synced to Entra astrustType: ServerAd,dsregcmdshows AzureAdJoined+DomainJoined YES, pilot.test getsAzureAdPrt: YES. On hybrid-joined machinesNgc PreReqResult: WillNotProvision(PolicyEnabled NO) -> Windows Hello does not auto-provision (no Hello popup) -- exactly what shared caregiver devices need, so no separate Hello-disable step. - Device control is one-at-a-time: caregiver machine computer objects are moved into
OU=Caregiver Devices(only that OU is in sync scope) and into a location groupSG-PC-MainTowerorSG-PC-MemoryCare. Add a device = move it into the OU + correct location group. - App + printer delivery GPO
CSC - Caregiver Workstation({3B5CD9A6-A278-4676-A9FD-9396D21A8261}, User-config GPP) -- BUILT + VALIDATED on NURSESTATION as pilot.test (2026-06-05). Linked atOU=Caregivers,OU=Departments; security filter =SG-Caregivers-Test(Apply, pilot.test only) + Authenticated Users (Read, for MS16-072). Go-live = swap filter toSG-Caregivers. Contents: 3 desktop shortcuts -- ALIS, LinkRx, Helpany (https://app.safe-living.com/login-- named "Helpany," the brand caregivers know) -- + 6\\CS-SERVERshared printers (NursesPrinter, HealthServices, MCMedTech, MCReception, MCDirector, CopyRoom) with default printer by device location (Nurses forSG-PC-MainTower, MC MedTech forSG-PC-MemoryCare, computer-context ILT) + HKCULegacyDefaultPrinterMode=1so the default sticks. Build scripts:clients/cascades-tucson/scripts/build-caregiver-gpo.ps1+link-caregiver-gpo.ps1. NOTE: the domain-wideCSC - Printer DeploymentGPO is intentionally disabled (empty CSE / version 0) and is not to be used -- reference only. - Device lockdown GPO
CSC - Caregiver Device Lockdown({E6174988-2721-4D96-ADF5-F5BB44E92769}, computer-only, linked toOU=Caregiver Devices) -- DEPLOYED 2026-06-05. Auto-logoff is a HIPAA requirement (SS164.312(a)(2)(iii)) for shared PHI devices. Settings: screen lock at 3 min, auto sign-out at 15 min total idle, 90-second warning before sign-out, never sleep (display off 10 min). Delivered via a computer startup script (caregiver-lockdown.ps1, in SYSVOL) that setsInactivityTimeoutSecs=180, powercfg, and registers a logon-triggered scheduled task running an idle monitor in each caregiver's session. Deploy script:deploy-device-lockdown-gpo.ps1. Startup scripts run at boot -- NURSESTATION must reboot to activate (not yet verified). Companion: ALIS app session timeout 20->15 min (Howard, ALIS admin) PENDING. Lock/logoff are device-level (affect any user on the device inOU=Caregiver Devices).
Status (as of 2026-06-30)
- Caregiver phone SSO -- Entra/identity side COMPLETE (group + Business Premium license + forced-change AD temp passwords for all 40). Remaining gate is the ALIS Email=UPN match (Howard) + creating ALIS records for the 7 with none + AD accounts for the 3 ALIS-only caregivers.
- Proven working end-to-end on a hybrid-joined desktop (NURSESTATION + pilot.test): caregiver lockdown (CA off-network block + device allow-list) and silent ALIS SSO. The allow-list policy
1b7fd025carries NURSESTATION's current deviceIdd3bf931f-f128-4261-8398-b46c34a4b342and the device is taggedextensionAttribute1=CSCCaregiverDevice. - GPOs DEPLOYED:
CSC - Caregiver Workstationbuilt and validated on pilot.test.CSC - Caregiver Device Lockdowndeployed toOU=Caregiver Devices2026-06-05. Go-live (still gated on all devices domain-ready): swap GPO filterSG-Caregivers-Test->SG-Caregivers; CA allow-list test group ->SG-Caregivers; move real caregiver machines intoOU=Caregiver Devices+ correctSG-PC-*location group one at a time. Still pending: lower ALIS app timeout 20->15 min; reboot NURSESTATION to verify lockdown. - Independent open item: Microsoft case for
INTUNE_A PendingInput-- does NOT block caregiver access (hybrid+GPO path replaces the Intune dependency).
Profile
- Contract type: Prepaid hour block
- Key contacts:
- Meredith Kuhn -- Assistant Manager (ASSISTMAN-PC); internal billing contact. NEVER set her as ticket contact in Syncro -- she is the wrong default that keeps being selected.
- John Trozzi -- Maintenance staff, Mac at 201cascades@gmail.com (shared facility account)
- Lauren Hasselman -- Accounting
- Zachary Nelson -- Accounting Assistant
- Lois Lane -- CareTakers department head (DESKTOP-KQSL232); resistant to domain migration; John Trozzi is liaison
- Crystal Rodriguez -- staff
- Sharon Edwards -- Life Enrichment Assistant (DESKTOP-DLTAGOI)
- Ashley Jensen -- Accountant (DESKTOP-U2DHAP0)
- Shelby Trozzi -- MemCare Director (MDIRECTOR-PC)
- Chris Knight -- Accounting / Business Office (same access tier as Lauren Hasselman); chris.knight@cascadestucson.com (alias: c.knight@cascadestucson.com). Workstation setup 2026-06-08: machine DESKTOP-N5G1ROO (Win 11 Pro for Workstations) domain-joined + GuruRMM-enrolled (agent
205025ee-2676-4498-8a27-e88562a6f69a), Office installed. AD accountchris.knight(OU=Administrative) finished to match Lauren. Mailbox remains cloud-only/unsynced (same split state as Lauren). - JD Martin -- Syncro-confirmed contact (jd.martin@cascadestucson.com); Chef's Office user on CHEF-PC (his default printer = USB Chef Printer).
- Lupe Sanchez -- staff (DESKTOP-TRCIEJA). EOL workstation (Gateway ZX6971 AIO, i3-2120, 8 GB RAM, Win11 unsupported). Decision 2026-06-18: replace machine (dual-AV + EOL hardware causing slow Excel; no remediation on current box). GuruRMM agent
c9bf1a2d-bfdc-401e-9cc8-f9e90bb19587(resolve live by hostname; UUIDs change on re-enroll). - Tamra Matthews -- Move-In Coordinator (Marketing/Sales), SALES4-PC. OFFBOARDED 2026-06-30 (left June 2026 -- see Access section).
- Syncro contact emails (authoritative): ashley.jensen@, jd.martin@, crystal.rodriguez@, John.trozzi@, meredith.kuhn@, accounting@/accountingassistant@cascadestucson.com.
- Billing rate: $175/hr all labor (prepaid block customer)
- Hours remaining: 37.5 hrs as of 2026-06-30 (live Syncro). Prior: 46.75 hrs as of 2026-06-26; 47.75 hrs as of 2026-06-25; 48.25 hrs as of 2026-06-24. Always live-check via
GET /customers/20149445before billing. - Syncro customer ID: 20149445
- Managed devices (Syncro assets): 29 (live 2026-06-30)
- Active tickets: 0 open Syncro tickets as of 2026-06-30 (live Syncro). See Active Work and session logs for ongoing project work (domain migration, EDR rollout, caregiver phone SSO, VLAN 20 printer migration).
- #110680053 / #32303 -- Entra / domain migration project. Status: Invoiced as of 2026-06-05. Plan:
C:\Users\Howard\.claude\plans\wise-discovering-panda.md - #109412123 -- Entra setup project (verify status)
- #32403 -- Meredith locked Word doc (0.5h remote, billed 2026-06-10, Invoiced)
- #32417 -- Shared mailboxes Grievances+Surveys (0.5h remote, billed 2026-06-12, Invoiced)
- #32444 -- Voice VLAN 30 + RF optimization (7h: 4 onsite + 3 remote, billed 2026-06-19, Invoiced)
- #110680053 / #32303 -- Entra / domain migration project. Status: Invoiced as of 2026-06-05. Plan:
Infrastructure
Servers & Services
| Host | IP | Role | OS | Notes |
|---|---|---|---|---|
| CS-SERVER | 192.168.2.254 | DC, DNS, DHCP (no scopes), File Server, Hyper-V host, Print Server | Windows Server 2019 Standard | Dell PowerEdge R610 (~2009 hardware, 16+ years old). Single DC -- CRITICAL risk. GuruRMM agent ID: c39f1de7-d5b6-45ae-b132-e06977ab1713 (re-enrolled; always resolve live by hostname, never hardcode the UUID). Clients reach SMB at 192.168.2.248 (registered DNS / Ethernet); .254 is the Hyper-V vEthernet NIC. RAID HEALTHY per live OMSA 2026-06-24 -- see hardware note below. All Datto software fully removed 2026-06-26 (was DattoAV/EDR/RMM leftover, not GravityZone). |
| CS-SERVER iDRAC | 192.168.2.65 | Out-of-band management | -- | Dell OOB interface |
| CS-QB (Hyper-V VM on CS-SERVER) | 192.168.2.228 | (label "VoIP server" -- STALE) | -- | 2026-06-16 recon: SMB/445 only, no SIP response -- NOT a live SIP PBX. Phones appear cloud-registered (Vertical). Label predates the wireless-phone transition; revisit/retire. |
| cascadesDS (Synology NAS) | 192.168.0.120 | NAS / legacy file storage | DSM 7.2.1-69057 | Port 5000 HTTP. Workgroup name is "CASCADES" -- same as AD short name, causing Kerberos auth failures from domain-joined machines. Slated to become backup-only. Synology Drive Server 3.5.0-26088 (active, port 6690 SSL). Current Drive sync: CS-SERVER Drive Client (v7.5.0.16085, runs as sysadmin) syncs Sync-user My Drive (/volume1/homes/Sync/Drive/) -> D:\Shares\Main (one-way download). Real shared folders (Server 1.9 G, Management 5.5 G, Public ~50 G, SalesDept ~23 G, etc.) are NOT in scope -- Team Folder migration pending. |
| pfSense Firewall | 192.168.0.1 | Perimeter firewall, inter-VLAN routing, DHCP/DNS | pfSense Plus 25.07-RELEASE | Netgate device. cert CN=pfSense-685f277aa6886. Dual-WAN. All DHCP (CS-SERVER DHCP role has no scopes). 199 DHCP subnets (per-unit /28 VLANs, assisted-living L2 isolation). SSH shell access works from onsite (no interactive menu) but tcp/22 is blocked from the OpenVPN subnet -- use the GUI (443) remotely. Admin vault: clients/cascades-tucson/pfsense-firewall. OpenVPN user Howard: vault clients/cascades-tucson/pfsense-openvpn-howard. Config vaulted 2026-06-17: clients/cascades-tucson/pfsense-config-backup-2026-06-17.sops.yaml. pfSense is ZFS (power-loss resilient). Logs are PLAIN TEXT (not clog). |
[CORRECTED 2026-06-24 -- LIVE OMSA] CS-SERVER RAID is HEALTHY, not degraded. Dell PowerEdge R610 (Service Tag 9MQFTK1), basic SAS 6/iR Integrated controller (3 Gbps, no cache), Status Ok. A live omreport query (Dell OMSA on CS-SERVER via RMM) shows both virtual disks Ok/Ready and all 5 physical disks Online/Ok, Failure Predicted: No, all LEDs green. The 2026-06-15 "degraded" state (PD 0:0:3 Critical/Removed) self-recovered -- the flaky consumer drive dropped out and re-synced after a power cycle (the ESM hardware log shows repeated drive remove/install events across the 6/17 + 6/23 outages). Do NOT pull a drive -- there is nothing failed to swap.
Live physical-disk map (OMSA, 2026-06-24):
| ID | Size/Type | Make / Serial | Role |
|---|---|---|---|
| 0:0:0 | 1.2 TB SAS | Seagate ST1200MM0088 / Z400WHK8 | VD0 (D:) mirror member, Online |
| 0:0:1 | 1.2 TB SAS | Seagate ST1200MM0088 / S400RL2N | VD0 (D:) mirror member, Online |
| 0:0:2 | 320 GB SATA | Hitachi HTS545032B9A300 / …1DR | VD2 (C:) mirror member, Online |
| 0:0:3 | 320 GB SATA | WDC WD3200BEVT / WD-WXEX08URD116 | VD2 (C:) mirror member, Online (the 6/15 flaky drive) |
| 1:0:4 | 1.2 TB SAS | Seagate ST1200MM0088 / Z400WHML | GLOBAL HOT SPARE (protects the 1.2 TB D: mirror; do NOT remove) |
- VD0 = D: (1,117 GB RAID-1) Ok; VD2 = C: (297.5 GB RAID-1) Ok. Windows sees only these 2 virtual disks.
- 1:0:4 is the GLOBAL HOT SPARE (not "unused") -- matched to the D: mirror, gives it auto-rebuild protection. Pulling it strips D:'s safety net. It cannot rebuild the 320 GB C: mirror (size mismatch), so the C: mirror has no spare.
- [WARN] PSU redundancy lost (ESM log "Power supply redundancy is lost") -- one of the dual PSUs isn't delivering; check cords/feeds/LEDs onsite.
- Planned (NOT emergency) reliability upgrade: replace the two consumer 320 GB drives (0:0:2 Hitachi + 0:0:3 WD, esp. the flaky WD) with 2x enterprise SATA SSD (already purchased) on a scheduled window with a confirmed image/system-state backup. DC migration off the 16-yr-old R610 remains the real long-term fix.
- LESSON: the prior "[CRITICAL] degraded -- replace drive" flag was a 9-day-stale snapshot; acting on it (SSDs purchased) before a live check was premature. Always pull live OMSA/iDRAC before drive action.
[INFO] Backup -- gap closed (2026-06-15); verified running 2026-06-24. Mike installed ACG cloud backup (MSP360/CloudBerry -> ACG-backup server) on CS-SERVER, addressing the longstanding SS164.308(a)(7) "no backup" HIPAA gap. Live check 2026-06-24: last run (6/24 00:10) = "Plan status: Success", 0 failed; 575.7 GB / 248k-file dataset already in the cloud (only 465 MB changed -> full baseline exists, incrementals working). Still to confirm: this looks FILE-LEVEL, not image/bare-metal/system-state -- for a DC that is a DR gap; confirm with Mike whether a separate image/system-state backup exists before treating it as full disaster coverage. Set/confirm retention.
[INFO] Endpoint security -- Datto EDR/AV is the ACG-managed stack (migration off Bitdefender substantially complete). Cascades migrated from Syncro-deployed Bitdefender GravityZone BEST to Datto EDR + Datto AV (Infocyte/azcomp4587.infocyte.com). Datto EDR org 2d5ea96e-3228-461b-9c60-13ae464b61d8, target group 1dbd2b02-f7df-45d0-a7f2-18667f48447f, reg key 6qw68y2rwl. State as of 2026-06-26: ~34 agents enrolled. Datto EDR installed on 7 machines 6/25 (org 27->34). Bitdefender REMOVED from RECEPTIONIST-PC (both physical boxes, serials MJ0KQH4R + MJ0KQHNP) via GravityZone console "Uninstall client" task -- the GravityZone Public API createUninstallTask is DEAD in this version and BEST anti-tampering (no uninstall password on policy "GPS Default") blocks endpoint-side uninstall, so removal is console-only. 6 orphaned C:\Program Files\Bitdefender folders deleted (safety-checked). RECEPTIONIST-PC is two distinct physical machines sharing a hostname -- dedup-by-hostname masks the second box. Straggler pass 2026-06-26: BD-checked reconnected machines -- DESKTOP-MD6UQI3, Laptop4, SALES4-PC all NO_BITDEFENDER (clean); DESKTOP-TRCIEJA = BD_ACTIVE (needs the GravityZone console uninstall; slated for replacement anyway). NurseAssist got its queued EDR install confirmed (agent 23c3c36e, AV on, v3.17.1.5552) after its Home->Pro upgrade. DESKTOP-F94M8UT (Alma Montt's, powered off) BD-aware EDR install queued (cmd a4623704, auto-fires on reconnect); DESKTOP-KQSL232 (Lois Lane's old box) decommissioned -- removed from list. CS-SERVER's endpoint AV was DattoAV, NOT GravityZone Bitdefender (see SMB/AV pattern) -- all Datto software fully removed from CS-SERVER 2026-06-26. Pending: DESKTOP-TRCIEJA GravityZone console uninstall; confirm Cascades removed from Syncro's Bitdefender deployment (GUI-only) so BD doesn't redeploy; GravityZone portal cleanup of stale RECEPTIONIST-PC endpoint records; reconcile laptop3 (EDR agent, no RMM agent); confirm/remove stale EDR agents laptop1 + cascades-laptop.
[WARN] Power outage (2026-06-17): Building power outage took the entire Cascades network down. Root cause: pfSense was plugged into the surge-only side of the UPS (no battery) -- it hard-powered-off uncleanly. ZFS survived. Dirty boot caused a duplicate dhcpd and a 2nd-floor switch (USL24PB, 192.168.2.193) with one-way L2 forwarding blocking DHCP OFFERs. Howard killed the duplicate dhcpd remotely; Mike re-seated pfSense onto battery outlets, restored config from on-box auto-backup (12:20 version, VLAN30 intact), reset+re-adopted Switch 2nd Floor #2. Network fully restored. Post-recovery casualties: devices that booted during DHCP-down window cached disconnected state (kitchen thermal printer fixed by power-cycle). Incident report: clients/cascades-tucson/reports/2026-06-17-power-outage-incident.md.
[INFO] Planned power outage (2026-06-23, 05:30-09:00 MST) -- clean shutdown executed: Building-wide electrical work scheduled a 3.5h power cut. To avoid a repeat of the 6/17 dirty-shutdown damage, all three core devices were armed the prior evening (2026-06-22 ~19:06) to shut THEMSELVES down on self-contained local schedules -- CS-SERVER (Windows task ACG-PlannedOutage-Shutdown -> stop CS-QB VM -> Stop-Computer, 05:28), Synology (/sbin/poweroff, 05:28), pfSense (shutdown -p now, 05:30) -- so they fired independent of any remote session or the OpenVPN tunnel, with the UPS carrying them through the 05:30 cut. Verified clean (2026-06-23 05:31 MST): CS-SERVER confirmed offline via GuruRMM cloud at last_seen 05:29:49 MST; pfSense + Synology unreachable as expected. Pre-flight (2026-06-22) verified: cloud backup last full SUCCESS @ 6/22 00:11 (0 errors); iDRAC AC Power Recovery ON + Synology auto-restart ON; John Trozzi onsite for physical power-on ~09:00. Bring-up bottom-up: pfSense first -> switches/APs re-adopt -> CS-SERVER -> Synology -> straggler sweep. Runbook: clients/cascades-tucson/docs/runbooks/2026-06-23-planned-power-outage.md.
Email & Identity
- M365 tenant: cascadestucson.com | Tenant ID:
207fa277-e9d8-4eb7-ada1-1064d2221498 - M365 license: Business Premium (SPB, SKU
cbdc14ab-d96c-4c30-b9f4-6ada7cdc1d46) -- 45 seats enabled / 45 consumed / 0 free (as of 2026-06-30; Howard bought 11 more seats on 6/30 and all 40 caregivers consumed seats -- was 34 enabled / 3 consumed). Business Premium includes Entra ID P1 (required per-user by the caregiver CA lockdown). Legacy Office 365 Business Standard (O365_BUSINESS_PREMIUM) -- SUSPENDED (0 enabled / 37 suspended). Relicensing the remaining suspended-Standard users onto Business Premium is pending. (Tamra Matthews' O365 Business Standard seat was freed on offboarding 2026-06-30; Alma Montt's SPB seat was freed 2026-06-25.) - On-prem AD domain: cascades.local | UPN suffix: cascadestucson.com (added 2026-04-13 for Entra Connect SSO readiness)
- MX / mail flow: Exchange Online (M365). SPF:
v=spf1 a mx ip4:72.194.62.5 include:spf.protection.outlook.com include:spf-0.secureserver.net -all. DKIM: both M365 selectors published. DMARC:p=quarantine;pct=100-- upgraded from p=none. Reports toinfo@cascadestucson.com(unmonitored). No third-party email gateway (EOP direct MX). - MFA: CA policy "Require MFA for all users" is enabled. Caregiver bypass in progress -- caregivers cannot satisfy MFA (no personal device), so three scoped CA policies use BLOCK instead. Voice-call MFA is disabled tenant-wide (SMS + Authenticator are the allowed methods). Exception: security group "MFA - Voice Call Scoped (sysadmin)" (id
304f941e-3594-4705-b8e6-ee676297df11, single membersysadmin@) has Voice method enabled. - Entra Connect: Installed on CS-SERVER 2026-04-25. Exited staging 2026-05-14 -- actively syncing (last sync confirmed 2026-05-27).
SG-Caregiversis an on-prem AD group synced via Entra Connect -- cloud/Graph group adds fail (HTTP 400); all membership writes must be done on-prem (CS-SERVER via RMM). OU=Administrative not yet in sync scope; UPN suffix updates for Administrative OU users pending before that OU can be added. - Break-glass accounts: Two planned (
breakglass1-csc@cascadestucson.com,breakglass2-csc@cascadestucson.com). Confirmed not yet created as of 2026-05-27. FIDO2 YubiKeys ordered -- arrival unconfirmed. - Admin accounts:
admin@cascadestucson.com-- Mike's working admin (cloud-only, Connect-excluded by design)sysadmin@cascadestucson.com-- Howard's working admin (cloud-only, Connect-excluded by design). Object id:471b13dc-3cf8-416b-a132-f5f3bc8d1cc8. Vaulted atclients/cascades-tucson/m365-sysadmin.sops.yaml.
- ALIS (clinical SaaS): https://cascadestucson.alisonline.com -- Entra SSO live and working. communityId 622. Install key:
d796539d-356b-4190-9c17-35f0f1129376. Vault:clients/cascades-tucson/alis-sso-app-registration.sops.yaml(API user cred:clients/cascades-tucson/alis-api-howard-user.sops.yaml-- username must be tenant-qualifiedhoward.enos@cascadestucson). ALIS application IDd5108493-cba8-4f08-90b6-1bb0bc09eb2a, client secret expires 2028-05-06 (rotation reminder -- expiry breaks ALIS SSO tenant-wide). Per-caregiver: ALIS staff-record Email must match Entra UPN exactly. Staff endpoints are read-only via API -- writes are done by uploading abuild-import.xls in the ALIS UI. BAA with Medtelligent not yet verified.- Admin consent (2026-06-03): Tenant-wide admin consent (
AllPrincipalsUser.Read) granted on ALIS Entra service principal (e1cae4ad-5beb-44ca-82d4-434c9bd835ad). This resolvedAADSTS65001sign-in failures. - How to enable ALIS SSO for one user: (1) Tenant-wide admin consent already done globally. (2) In ALIS admin -> Staff -> user's record, set Email = exact Entra UPN. (3) User signs in via "Sign in with Microsoft." (4) Turn off ALIS-native 2FA (Entra is the second factor; native 2FA conflicts and locked out Karen Rossini).
- Diagnostic signature: a user with zero ALIS-app sign-in events in Entra sign-in logs is still on the old direct-login path -- fix is the ALIS Email match, not anything in Entra.
- Admin consent (2026-06-03): Tenant-wide admin consent (
- Caregiver phones: 22 Samsung Galaxy A15s enrolled in Intune Shared Device Mode (SDM). Enrollment profile:
CSC - Android Shared Phones (Entra SDM)(9a0fcc6d); 25 devices enrolled per 2026-06-03 Intune pull. Dynamic group:Cascades - Shared Phones(ea96f4b7). Android enrollment token expires 2027-05-08 -- expiry does NOT unenroll existing devices. Caregiver identities now licensed + grouped + temp-passworded for phone SSO (2026-06-30) -- see Entra Access Architecture. - Audit retention: Approved 2026-04-29. Azure Log Analytics (90d) + Storage Account (6yr) in ACG subscription
e507e953-2ce9-4887-ba96-9b654f7d3267, RGrg-audit-cascadestucson. Not yet built. - Inky: No Inky deployment exists in this tenant. Confirmed 2026-06-04.
- EXO MSP app auth note (2026-06-04): When the MSP app cert is not in the Windows cert store, use client_credentials flow to obtain an EXO-scoped access token and connect via
Connect-ExchangeOnline -AccessToken. App: ComputerGuru Exchange Operator (b43e7342-5b4b-492f-890f-bb5a4f7f40e9). Vault:msp-tools/computerguru-exchange-operator.sops.yaml. - Shared mailboxes (created 2026-06-12):
grievances@cascadestucson.comandSurveys@cascadestucson.com-- both SharedMailbox type, cloud-only, no license consumed. Delegated to Meredith Kuhn and Ashley Jensen with FullAccess (auto-mapping) + SendAs on each. All 8 permission grants verified. Ticket #32417.
Network
- ISP / WAN: Dual-WAN Cox. WAN1 igc0
184.191.143.62/30(Cox Fiber, primary, gateway184.191.143.61) + WAN2 igc372.211.21.217/27(Cox Coax, secondary, static);WAN_Groupgateway group; both active full-duplex, no loss events (verified 2026-06-16). Both WAN IPs added as Cascades Named Location in Entra (ID:061c6b06-b980-40de-bff9-6a50a4071f6f). Measured bandwidth (2026-06-18): WAN1 fiber upload ~522 Mbps; RRD 3-day peaks ~680 Mbps down / 98 Mbps up (actual usage). WAN2 coax upload unmeasured (remote source-route test failed -- needs a WAN2-routed host or the Cox bill). 30 calls ~= 3 Mbps vs ~522 Mbps fiber headroom -> the WAN is NOT the everyday voice bottleneck (RF is); voice QoS is insurance for WAN2 failover + rare WAN1 saturation. - Firewall: pfSense Plus 25.07-RELEASE (Netgate) at
192.168.0.1, cert CN=pfSense-685f277aa6886. Admin vault:clients/cascades-tucson/pfsense-firewall. SSH shell access works onsite (no interactive menu) but tcp/22 is blocked from the OpenVPN subnet -- use the GUI (443) remotely (the unifi-wifipfsense-ssh.shskill silently returns empty over VPN). OpenVPN user Howard: vaultclients/cascades-tucson/pfsense-openvpn-howard(split-tunnel;route 192.168.0.0/22; use OpenVPN GUI or OpenVPN Connect with DCO disabled for stability). Logs are PLAIN TEXT on 25.07 -- read with tail/grep, NOT clog. pfSense has an OpenVPN--inactiveidle timeout (~300s) on the server; it disconnects clients after ~5 min of no tunnel data (keepalive pings do NOT reset this counter). Fix proposed 2026-06-18; not applied. [ROUTING GOTCHA 2026-06-30] The LAN "Default allow LAN to any" rule has Gateway = WAN_Group (dual-WAN policy routing), which shoves LAN->internal-VLAN traffic (e.g. CS-SERVER -> VLAN 20 printers) OUT THE WAN. This is NOT a firewall block. Fixed with a top LAN pass rule: source CS-SERVER192.168.2.248, dest10.0.20.0/24, Gateway = default -- restores full server access to VLAN 20 (printers etc.) without matching resident/guest ingress. [OUTAGE 2026-06-17] pfSense was on UPS surge-only side -- moved to battery-backed outlets by Mike. On-box auto-backup restored; config vaulted. Enable Netgate AutoConfigBackup to prevent future off-box gap.- [INFO] pfSense health check (2026-06-16): gateway ruled out as WiFi factor -- DHCP not exhausted, unbound DNS up, both WANs full-duplex/stable, firewall states 28-31k/790k, load 0.6.
- LAN / VLAN layout: Primary staff/AP network
192.168.0.0/22(pfSense .0.1, cascadesDS .0.120, UniFi APs + most WiFi clients on 192.168.2.x/3.x). DHCP pool 192.168.2.2-192.168.3.254 (~507 cap, ~270 active ~53%). Per-unit /28 VLANs: 199 DHCP subnets total, mostly10.x.y.0/28per apartment (assisted-living L2 isolation) + Staff/Internal VLAN 20 ("CSCNET",10.0.20.0/24, gw10.0.20.1-- the target VLAN staff machines + printers are being migrated onto, off the flat old LAN "CSC ENT" 192.168.0.0/22) + Guest VLAN 50 (10.0.50.0/24, RFC1918 blocked) + Voice VLAN 30 (10.0.30.0/24, gw10.0.30.1). DHCP backend: ISC (Kea config present, dormant). Unbound DNS. - Switching: Full UniFi. 77 U7-Pro APs + 12 managed switches (1st Floor USW-48 PoE core; floors 2-4 USW-Pro-24-PoE; MemCare USW-Pro-24-PoE; USW Lite 8 PoE; USW-16-PoE VoIP switch). [WARN] ~25 switch ports linked at 100 Mbps but gig-capable (systematic cabling/NIC issue, 1st/2nd/3rd-floor switches; investigate after WiFi Phase A). 3 offline switches: Switch 2nd Floor #2, Switch 4th Floor #2, USW Pro Max 16. PoE budgets healthy. Port p38 (1st Floor USW) 4.0% tx-drop rate. All managed on the shared UOS controller (172.16.3.29, HTTPS 11443; see uos-server); Cascades site short name
va6iba3v, site_id685f39068e65331c46ef6dd2. Mesh topology: 2nd Floor Atrium is wireless-mesh parent for CC Bridge + salon (5 GHz backhaul ch36); 206 U7 Pro carries AP 108. Note: Switch 2nd Floor #2 (USL24PB, 192.168.2.193) was reset+re-adopted after the 2026-06-17 power outage. - WiFi SSIDs:
- CSCNet -- shared PPSK SSID.
private_preshared_keys_enabled; ~230-242 per-key->network mappings (most keys -> per-room resident VLANs 101-631; a few -> Default; one phone key -> Internal/VLAN 20; one voice PPSK -> VOICE/VLAN 30). ~1,190 historical clients (residents' IoT/TVs, staff, phones). Do NOT repoint the SSID to move a subset of clients -- move at the PPSK level. wlanconf685f39078e65331c46ef7ee5; cred vaultclients/cascades-tucson/wifi-cscnet.sops.yaml. - CSC ENT -- legacy SSID, main LAN (192.168.0.0/22), being deprecated as migration proceeds (planned to become the permanent 5 GHz-only WPA2 "device island" for Poly voice + Helpany sensors -- see VoIP/network section)
- Guest -- isolated, VLAN 50
- CSCNet -- shared PPSK SSID.
- Wireless RF status (applied 2026-06-19 -- ~587 concurrent clients):
- 2.4 GHz is the primary pain band: avg TX-retry ~10%, cu_total 69-94% live, catastrophic external neighbor BSSID density (ch6 ~33k BSSIDs, ch1 ~19k, ch11 ~17k). 27 of the 40 worst clients on 2.4 GHz (retry 11-42%), mostly IoT/legacy. Root cause: extreme radio density; external saturation limits benefit of any 1/6/11 channel re-plan.
- 2.4 GHz power -> MEDIUM (APPLIED 2026-06-19, validated, kept): 42 over-thinned-
lowradios floors 1-4 + 5 MemCare floors-5/6auto/full radios (505/517/608/615/622) -> Medium. 24 thinned-disabled radios stayed disabled; 5 mesh-auto APs untouched. Non-regressive. Corrected the 2026-06-17 over-thinning regression (retry had risen to 23.4% at Low). Remaining 2.4 levers (deferred): min-RSSI on 6 APs currently OFF (615, 608, 505, 517, 622, salon); 1/6/11 channel re-plan (marginal against external density). - 5 GHz: clean DFS 40 MHz plan APPLIED (2026-06-19) -- retry HALVED. 72 non-mesh APs on channels {52,60,100,108,116,124,132,140}, 0 co-channel pairs. Result: retry 8.7->3.8 avg (median 8.2->2.1, ~half). Validated: all 72 APs holding DFS, 0 radar vacates, satisfaction median 99. Mesh APs (2nd Floor Atrium, CC Bridge, salon, 108) left on auto. DFS channels at this site are 4-5x cleaner than non-DFS (2-3% busy vs ch149=12%, ch157=28%, ch44=22%); the earlier "non-DFS only" recommendation was reversed by measured survey data (74/74 APs).
dfs-check.sh2026-06-16 + nightly: ZERO real radar events fleet-wide. Safety net: UniFi auto-vacates one AP on a radar hit; follow-up = recurringdfs-check.shmonitor. - 6 GHz BLOCKED on CSCNet: 75 radios active but only ~1 client. Root cause: CSCNet
wlan_bands=[2g,5g](not broadcasting 6g); enabling 6 GHz on WPA2/PPSK SSID requires WPA3+PMF conversion (Wpa3MandatoryFor6GHzBand), touching all 427 clients. Largest untapped clean capacity. Deferred to Howard's supervised decision on the SSID security conversion. - CSCNet BSS-transition (802.11v): ON (applied 2026-06-19).
- 3 AM AP auto-upgrade: OFF (left off after overnight run; re-enable when ready).
- AP 103 saturated (5 GHz): ch149 at 75% airtime, ~25,900 retries, 12 clients -- was Lauren's locked AP. With the DFS 40MHz plan applied, ch149 is now assigned elsewhere on the fleet; AP 103's channel should have changed. Verify load post-settle.
- AP-level satisfaction 95-100 fleet-wide. Pain is in the client tail (IoT stuck on 2.4).
- AP 108 (Floor 1) offline pending a new cable run. Stale duplicate controller object ("108" vs "108U7 Pro") to clean up separately.
- VoIP (vendor: Vertical -- Richard Turner RTurner@vertical.com): Two phone fleets -- 8 AudioCodes (OUI
00:90:8f, WIRED on USW-16-PoE ports 1-8, externally powered / PoE OFF) and Poly (OUI48:25:67, WiFi via CSCNet PPSK) -- 28 active (29 re-keyed 2026-06-19, 1 removed bad). All on VOICE VLAN 30: 28 Poly + 8 AudioCodes (.224-.231) + Vertical desktop (.201) = 37 devices. Phones mark DSCP EF (46). [2026-06-19 hardware change] John (Trozzi) reported the Kitchen server phone (48:25:67:64:95:7a) BAD and pulled it; the Bistro phone (.236,48:25:67:64:94:84) was relocated to the Kitchen to cover it -- so the BISTRO now has NO phone (replacement pending, set up + re-key when it arrives). (Verify VLAN via the clientvlanfield, NOT the cached display IP.) The Vertical-Remote management desktop (10.0.30.201, MACe4:e7:49:52:3a:06, WIRED USW-16-PoE port 16, VOICE VLAN 30, DHCP -- confirmed not static, LogMeIn remote access, no pfSense OpenVPN) is live on VLAN 30. No on-prem SIP PBX found -> phones appear to register to a cloud/hosted PBX (Vertical). - [2026-06-19 COMPLETE] Voice VLAN (VLAN 30) consolidation: dedicated isolated VLAN 30 VOICE (
10.0.30.0/24, gw10.0.30.1, pfSense igc1.30, DHCP.100-.250, DNS8.8.8.8/1.1.1.1) holding ALL phones + the Vertical desktop; internet/cloud-PBX egress only, firewalled off VLAN 20 / main LAN / PHI / mgmt (HIPAA). Voice PPSK key on CSCNet -> VOICE: vaultedclients/cascades-tucson/wifi-voice-ppsk. Migration COMPLETE 2026-06-19: 37 devices on VOICE. Live inventory:docs/network/voice-phone-inventory.md.- Quality caveat + the actual fix (2026-06-19): the VLAN move does NOT by itself fix call quality. Per-phone re-look found residual dropped-calls are a band-selection problem, not RF/coverage -- several Poly handsets sit on saturated 2.4 GHz despite EXCELLENT 5 GHz-capable signal (-50 to -60 dBm, 36-96% retry), and controller band-steering (
no2ghz_oui, already ON) is NOT holding the Poly OUI on 5 GHz. The fix is a dedicated 5 GHz network, not phone-side band pinning -- Richard Turner (Vertical/Poly, 2026-06-24) confirmed Poly phones cannot be statically assigned to a band; Poly recommends a separate 5 GHz SSID (or disabling band steering on a shared SSID). - [PLAN 2026-06-24] CSC ENT device-island consolidation (Howard + Mike): the phone 5 GHz fix is now merged with the Helpany sensor rollout into one plan -- repurpose the existing CSC ENT SSID as a 5 GHz-only WPA2 PPSK "device island" carrying BOTH the Poly voice handsets (PPSK key -> VLAN 30) and the Helpany "Paul" radar sensors (PPSK key -> new VLAN 40), separated at the VLAN layer. Both vendors transition their devices remotely once we hand them the network. Helpany is WPA2-only (no WPA3/hybrid) and the Pauls are already on CSC ENT (key
Ftfd85710#), so they are not reprogrammed -- only band-moved; the phones get a new voice key. Onsite gate: verify per-room 5 GHz coverage before the band flip (steel walls; weak-5 GHz devices stay on 2.4 per both vendors' warning). CSC ENT is NOT deleted -- it becomes the permanent WPA2 island, which is the prerequisite that later lets CSCNet move to WPA3/WiFi7/6 GHz (that step is separately gated by the ~230 resident 2.4-only/WPA2-only IoT clients, NOT by the voice/sensor gear). Full design + sequence:docs/network/csc-ent-device-island-plan.md; folded intodocs/REMAINING-WORK-PLAN.mdWorkstream 6. - Full runbook:
clients/cascades-tucson/docs/network/voice-vlan-cutover.md. Voice-quality diagnostic:reports/2026-06-18-voice-quality-diagnostic.md. Holistic optimization plan:docs/network/network-optimization-master-plan.md; voice QoS design:docs/network/phase1-voice-qos-design.md.
- Quality caveat + the actual fix (2026-06-19): the VLAN move does NOT by itself fix call quality. Per-phone re-look found residual dropped-calls are a band-selection problem, not RF/coverage -- several Poly handsets sit on saturated 2.4 GHz despite EXCELLENT 5 GHz-capable signal (-50 to -60 dBm, 36-96% retry), and controller band-steering (
External Vendors & Mail Senders
- Helpany (resident safety sensors -- Sandro Cilurzo CEO / Eugenie Nicoud COO): "Paul" devices are ceiling-mounted radar fall/motion sensors (Sedimentum backend) -- no camera, no microphone (despite being colloquially called "IR cameras"). WiFi: WPA2-only, NOT WPA3/hybrid; 5 GHz-capable. Currently on SSID CSC ENT (key
Ftfd85710#), being moved to 5 GHz (see CSC ENT device-island plan in the VoIP/network section). Bandwidth negligible: <0.04 Mbps/device, fleet peak ~1.35 Mbps. Egress to*.sedimentum.com(5671 AMQPS, 8883 MQTT, 8030 HTTP, 443) + snapcraft/ubuntu (443). Helpany transitions devices remotely (engineering); key rotation needs 72 h notice + new key; reprogramming offline devices is hard. Rolled out floor-by-floor from 2026-06 (first shipment floors 1-2). Caregiver-facing app = app.safe-living.com (branded "Helpany"). Facility liaison: John Trozzi. - bill.com (BILL): Sends from
inform.bill.com,hq.bill.com,hello.bill.com,mc.bill.com. MX via pphosted.com (Proofpoint). Confirmed delivering successfully to meredith.kuhn, ashley.jensen, lauren.hasselman, zachary.nelson as of 2026-06-04. Safe sender:account-services@inform.bill.com. - BOK Financial: Sends from
bokfinancial.com. MX via pphosted.com (Proofpoint). DMARC p=reject. Zero emails to any cascadestucson.com user in 90-day history as of 2026-06-04 (likely wrong recipient address on BOK's side for the accounts in question).
Business Applications & Reporting Systems
Cascades' line-of-business / reporting SaaS (the systems they pull data OUT of, per Ashley Jensen 2026-06-17). Most are niche senior-living products:
| System | Function | Data-out path |
|---|---|---|
| ALIS (Medtelligent) | Clinical EHR (census/clinical) | Vendor reporting/export; API TBD. HIPAA -- BAA required before PHI leaves it. Their most important source. SSO live (see Entra section). |
| QuickBooks | Accounting | QBO = API + connectors; Desktop = ODBC |
| Bill.com | AP/AR | REST API (most automatable) -- see mail-sender note above |
| Relias | Training / LMS | Reporting export / API (completion data) |
| You've Got Leads | Senior-living CRM | Reporting/export; API varies |
| TELS (Direct Supply) | Facilities management | Reporting export; API uncertain |
| Focus HR | HR / payroll | Export or vendor API (plan-dependent) |
| Helpany (app.safe-living.com) | Caregiver app | Niche -- likely export-only |
| POS | Point of sale | Product TBD |
- [PROPOSED] Unified KPI dashboard (Ashley Jensen request, 2026-06-17): single dashboard pulling KPIs across the systems above. Recommended path: Phase 1 scheduled CSV/Excel exports -> SharePoint -> Power BI Pro dashboard; Phase 2 automate the API-capable systems (Bill.com, QuickBooks Online) via Power Automate. Power BI on-prem Gateway is the WRONG frame (bridges only on-prem DBs, not cloud SaaS). Internal scoping:
clients/cascades-tucson/docs/proposals/kpi-dashboard.md; client one-pager:.../kpi-dashboard-onepager.md. Status: parked, awaiting Ashley's day-one KPIs + freshness need + POS/Focus-HR specifics + ALIS analytics availability.
Access
- CS-SERVER: Via ScreenConnect or GuruRMM (live agent ID
c39f1de7-d5b6-45ae-b132-e06977ab1713as of 2026-06-30; re-enrolls -- resolve live by hostname, do not hardcode). Vaulted domain-admin cred:clients/cascades-tucson/cs-server.sops.yaml(usernamesysadmin). - CS-SERVER iDRAC: 192.168.2.65
- pfSense admin (HTTPS): https://192.168.0.1 -- vault:
clients/cascades-tucson/pfsense-firewall.sops.yaml - pfSense SSH:
ssh admin@192.168.0.1(system OpenSSH; drops to shell directly, no interactive menu) -- tcp/22 blocked from the OpenVPN subnet; use the GUI remotely -- vault admin cred:clients/cascades-tucson/pfsense-firewall.sops.yaml; pfsense-ssh.sh (unifi-wifi skill) for scripted access onsite. - pfSense OpenVPN (Howard): split-tunnel; vault:
clients/cascades-tucson/pfsense-openvpn-howard.sops.yaml(userHoward; route 192.168.0.0/22). Use OpenVPN GUI or OpenVPN Connect with DCO disabled for stability. Howard-Home is 10.137.42.0/24 (renumbered 2026-06-16). Server has a configured--inactiveidle timeout (~300s) that silently drops idle clients. - pfSense config backup (2026-06-17):
clients/cascades-tucson/pfsense-config-backup-2026-06-17.sops.yaml - Synology DSM: http://192.168.0.120:5000 -- vault:
clients/cascades-tucson/synology-cascadesds.sops.yaml(admin). Drive Server port 6690 (SSL). [SECURITY] Synology Cloud Signin Portal credential (clients/cascades-tucson/synology-signin-portal.sops.yaml) was committed plaintext at vault commit 1fbc0e1 -- exposed in git history; encrypted go-forward but credential should be rotated. - M365 admin: admin@cascadestucson.com -- vault:
clients/cascades-tucson/m365-admin.sops.yaml - M365 sysadmin: sysadmin@cascadestucson.com -- vault:
clients/cascades-tucson/m365-sysadmin.sops.yaml - WiFi CSCNet: vault:
clients/cascades-tucson/wifi-cscnet.sops.yaml - WiFi Voice PPSK (VLAN 30): vault:
clients/cascades-tucson/wifi-voice-ppsk.sops.yaml - MDM service account: vault:
clients/cascades-tucson/mdm-service-account.sops.yaml - svc-scan (scan-to-folder service account): vault:
clients/cascades-tucson/svc-scan.sops.yaml. AD account on CS-SERVER for the Accounting Brother's SMB scans. - ALIS SSO app registration: vault:
clients/cascades-tucson/alis-sso-app-registration.sops.yaml; ALIS API user:clients/cascades-tucson/alis-api-howard-user.sops.yaml. - Caregiver AD temp passwords (2026-06-30): vault:
clients/cascades-tucson/caregiver-temp-passwords-2026-06-30.sops.yaml(40 caregiver AD temp passwords, all forced-change-at-next-login; keys = sAMAccountName). Hybrid PHS -> these are also the M365/phone sign-in. Retrieve withvault get(NOT get-field -- dotted keys). Do NOT inline any values. - UOS controller SSH (root): vault:
infrastructure/uos-server-ssh-key-- SSH/Mongo access forunifi-wifiskill anduos-mongo.sh. Vaulted 2026-06-15 by Mike. - UOS controller RW admin (Network API): vault:
infrastructure/uos-server-network-api-rw-- required to apply any radio/config changes. Vaulted 2026-06-15 by Mike. - UniFi AP device auth (Cascades): vault:
clients/cascades-tucson/unifi-ap-ssh-- direct AP SSH via site VPN (needed forwatch-ap.shlive stream; L3 reach to 192.168.2.x/3.x via split-tunnel VPN). Vaulted 2026-06-15 by Mike. - UOS controller (HTTPS): https://172.16.3.29:11443 (HTTPS 11443, not 8443) -- site
va6iba3v/ site_id685f39068e65331c46ef6dd2 - GuruRMM -- RECEPTIONIST-PC (frontdesk box, serial MJ0KQHNP): agent
2e8d8b73-82f6-4151-a3ce-879c55de4b82(front desk). The Memory Care box (serial MJ0KQH4R, agent57f19e17-8792-46cc-b9fd-f1909836cd17) shares the hostname -- rename to MEMCARE-STATION STAGED 2026-06-30 (applies on next reboot). - GuruRMM -- ASSISTMAN-PC (Meredith Kuhn): agent ID
cf86fa5e-96a2-494d-9cb1-8be22a518ad0 - GuruRMM -- DESKTOP-TRCIEJA (Lupe Sanchez): agent ID
c9bf1a2d-bfdc-401e-9cc8-f9e90bb19587(resolve live by hostname; UUIDs change on re-enroll) - Remediation tool: Full tiered app suite consented 2026-04-21. All six apps active: Security Investigator (
bfbc12a4), Exchange Operator (b43e7342), User Manager (64fac46b), Tenant Admin (709e6eed), Defender Add-on, Intune Manager. - [SECURITY -- OPEN 2026-06-25] Tenant Admin SP holds a STANDING Privileged Authentication Administrator (PAA) role. During Alma Montt's offboarding the
ComputerGuru - Tenant AdminSP was JIT-elevated to PAA to reset her password; Graph then blocked the automatic teardown ("removing self from built-in role is not allowed"), leaving the role assigned. Needs a Global Admin to remove in Entra (Roles & admins -> Privileged Authentication Administrator -> remove the SP); leave its standing Conditional Access Administrator role (intentional). Pending Mike's decision (coord message sent 2026-06-25). Recommended posture: keep JIT, fix the teardown so resets stop stranding PAA. - Alma Montt -- OFFBOARDED 2026-06-25 (terminated; MC Life Enrichment; no PHI/ALIS access). M365 sign-in blocked, 0 licenses, mailbox -> SharedMailbox (Shelby Trozzi FullAccess+AutoMap), hidden from GAL, groups removed; on-prem AD disabled + moved to
OU=Excluded-From-Sync. No litigation hold (no PHI). Verified live end-to-end and reconciled out of all active plans/rosters. Emergency password: vaultclients/cascades-tucson/alma-montt(do NOT re-enable without authorization). Record:docs/security/offboarding-2026-06-25-alma-montt.md. - Tamra Matthews -- OFFBOARDED 2026-06-30 (Move-In Coordinator, Marketing/Sales; left June 2026). M365 object
2d9cf0d1-2b0b-424e-9cd1-91eaac408837was cloud-only (single O365 Business Standard license) -- sessions revoked, sign-in blocked, password reset+vaulted, mailbox -> SharedMailbox (Crystal Rodriguez, Megan Hiatt, Meredith Kuhn, Ashley Jensen all FullAccess+AutoMap), hidden from GAL, license removed (seat freed), removed fromSales/All Cascades/SG-SSPR-Eligible. On-premTamra.Matthews(separate non-synced object in OU=Marketing) disabled + moved toOU=Excluded-From-Sync. No litigation hold despite PHI-adjacent (resident-intake) role -- Howard authorized the Alma-style preserve-but-no-hold posture (documented in the record; hold remains available -- tenant has Exchange Plan 2). Emergency passwords: vaultclients/cascades-tucson/tamra-matthews(m365_password + ad_password; do NOT re-enable without authorization). Record:docs/security/offboarding-2026-06-30-tamra-matthews.md. ALIS profile disable pending (Howard, if she had one). Follow-up: breach check on Megan Hiatt (April CREDENTIAL_STUFFING marker -- verify the April remediation held). - ComputerGuru Exchange Operator MSP app:
b43e7342-5b4b-492f-890f-bb5a4f7f40e9-- vault:msp-tools/computerguru-exchange-operator.sops.yaml. - Vault root:
clients/cascades-tucson/in vault repo
Patterns & Known Issues
Syncro / Billing
- Never set a contact on any Syncro ticket unless explicitly requested. At Cascades, Meredith Kuhn is the recurring wrong default that Syncro pre-selects -- she is not the correct contact. Leave
contact_idblank. Source:feedback_syncro_blank_contact.md. - Billing product for prepaid block draw: Use a real labor type (Remote, Onsite, etc.) -- NOT "Prepaid project labor" (exempt, won't decrement the block).
- Always live-check hours before billing:
GET /customers/20149445in Syncro. Treat all cached hour counts as approximate. - Windows Home->Pro upgrade billing: Syncro product "Windows Pro Upgrade" id 23571919 ($99/machine); machine-named lines + optional labor. MAK: vault
infrastructure/windows-pro-mak.sops.yaml.
Exchange Online / Message Tracing
- Get-MessageTrace is hard-deprecated (Sept 2025). Use
Get-MessageTraceV2instead. Key parameter change: useResultSize(notPageSize). The deprecation error may be silently swallowed by downstream jq filters -- if a trace returns unexpectedly empty, check the raw response for a deprecation error string before assuming no mail. - Sender-side suppression (SendGrid ESP): If a user never receives mail from a specific sender despite a healthy mailbox, and message trace shows zero records (not even bounces), consider a SendGrid suppression list. Fix requires contacting the sender's support -- there is no M365 action. Confirmed with bill.com / inform.bill.com.
- AutoMapping rolls back on rapid sequential grants. Granting FullAccess with
AutoMapping:$trueto several delegates in a rapid loop can silently lose the middle writes -- they contend on the samemsExchDelegateListLinkmultivalued attribute on the target mailbox and a failed link-write aborts the wholeAdd-MailboxPermission. Fix: grant one user at a time, spaced ~6-8s,Remove-MailboxPermission-> poll-until-NONE ->Add-MailboxPermission -AutoMapping $true-> verify-persists. The REST Exchange Operator schema does not exposemsExchDelegateListLink, so use grant-persistence as the success signal. (Tamra Matthews offboarding 2026-06-30.)
Active Directory / User Management
-
Security group assignment is always explicit. When creating or adding any Cascades user, always ask which security group(s). OU -> group auto-mirror was explicitly declined 2026-05-14.
-
SG-Caregivers= frontline caregivers ONLY (Howard 2026-06-30). Admin-adjacent staff (e.g. Veronica Feller in OU=Care-Assisted Living, Christine Nyanzunda in OU=Care-Memorycare),pilot.test, and disabled accounts are EXCLUDED -- this reverses the earlier 6/4 plan to add Feller/Nyanzunda. -
SG-Caregiversis an on-prem AD group synced by Entra Connect. Cloud/Graph adds fail (HTTP 400). Do all membership writes on CS-SERVER via RMM (Add-ADGroupMember). -
New user mandatory order (folder redirection):
- Create AD user
- Run
New-HomeFolder -Username "<sam>"on CS-SERVER (creates root + Desktop/Documents/Downloads/Music/Pictures with correct ACL) - Add to SG-FolderRedirect
- THEN first domain logon
- Skipping step 2 causes fdeploy to cache a failure silently and never retry.
-
Folder redirect recovery: If fdeploy cached a failure ("No changes detected"), run
clients/cascades-tucson/scripts/fix-shell-redirect.ps1via GuruRMM while user is logged in. Must set both GUID-based and legacy-name registry keys. Folders must already exist on server. -
fdeploy1.ini flags: Changed from
Flags=1211(includedGrant Exclusive Rightsbit 0x400, causing WRITE_DAC failures on new subfolders) toFlags=187. -
[ROOT CAUSE + FIX 2026-06-08] Native Folder Redirection was DOA on every machine -- the config file was MISNAMED. Root cause: the redirect targets in GPO
CSC - Folder Redirection({512B43A4-...}) were saved in a file namedfdeploy1.ini, but the Windows Folder Redirection CSE only ever readsfdeploy.ini. Fix: wrote a correctfdeploy.ini(5 folders,Flags=187,FullPath=\\CS-SERVER\Homes\%USERNAME%\<Folder>) into{512B43A4-...}\User\Documents & Settings\, bumped the GPO version 917506->983042 (GPT.INI and ADversionNumberkept in sync). Native FR now redirects all 5 folders on first logon.- LE GPO also broken:
CSC - Folder Redirection (LE)({889BE7BE-...}, linked at OU=Life Enrichment) has a completely empty\Usertree. Sharon Edwards / Susan Hicks have likewise only ever worked via the registry workaround. Follow-up: retire the LE GPO and put LE users intoSG-FolderRedirect, or apply the samefdeploy.inifix to the LE GPO.
- LE GPO also broken:
-
Login-screen hide (SpecialAccounts\UserList): An enabled local admin that does not appear in the Windows sign-in picker is a
SpecialAccounts\UserListsuppression, not a disabled account. Registry path:HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\SpecialAccounts\UserList, value<username>=0. Fix: delete the DWORD value.
File Shares & Scan-to-Folder (Accounting)
- Accounting department folder + scan dropbox (built 2026-06-09):
D:\Shares\Accountingon CS-SERVER -- inheritance broken; SYSTEM / BUILTIN\Administrators = Full;lauren.hasselman,chris.knight,zachary.nelson= Modify (no Everyone). Shared as\\CS-SERVER\AcctDept(Change: those 3 users +svc-scan; Full: Admins).- Share is named
AcctDept, NOTAccounting-- a printer share namedAccounting(Canon MF455DW,LocalsplOnly) already exists. Do not collide with it. svc-scan= dedicated AD service account (CN=Users, PasswordNeverExpires, CannotChangePassword) for the Brother's SMB auth. Vault:clients/cascades-tucson/svc-scan.sops.yaml.- REUSE
svc-scanfor EVERY future scanner->network-folder setup at Cascades (Howard, 2026-06-09) -- do NOT create a per-printer/per-folder scan account.
- Brother MFC-L8900CDW "Business Office" printer (10.0.20.220) -- Scan-to-Network profile (working 2026-06-09): Network Folder Path
\\192.168.2.254\AcctDept\Scans; Auth Method NTLMv2 (not Auto/Kerberos -- printer can't KDC across VLAN); Usernamecascades\svc-scan; PDF Multi-Page. - [CORRECTED 2026-06-24, live] CS-SERVER CAN reach VLAN 20 -- server-hosted printing to VLAN-20 printers works. CS-SERVER routes to
10.0.20.0/24via the default gateway (pfSense192.168.0.1) and pings the VLAN-20 gateway10.0.20.1fine. The VLAN-20 print queues already on the server print through it. Caveat: the printers often don't answer ICMP ping when asleep (and 9100 may show closed while idle) -- that is NOT a firewall block; a real print job wakes them. [REFINED 2026-06-30] The one real block was policy-routing, not the firewall: the LAN "allow LAN to any" rule's Gateway=WAN_Group sent CS-SERVER->VLAN 20 out the WAN. A top LAN pass rule (src 192.168.2.248, dst 10.0.20.0/24, gw=default) fixed it -- see pfSense Operations. - Persistent drive maps to
\\cs-server\AcctDept: Chris (DESKTOP-N5G1ROO) Y:, Zachary (ACCT2-PC) Y:, Lauren (DESKTOP-H6QHRR7) X:. \\CS-SERVER\BusinessOffice(Business Office - Brother L8900CDW,10.0.20.220) = the "Accounting Assistant" printer in room 101 -- one physical L8900CDW, already a shared print queue on CS-SERVER. Attached to Chris Knight's PC (DESKTOP-N5G1ROO) 2026-06-24. Do NOT create a duplicate "Accounting Assistant Printer" queue -- it's this one.- Executive restricted share (built 2026-06-24, ticket #32193):
D:\Shares\Executiveon CS-SERVER, shared as\\cs-server\Executive; inheritance broken; SYSTEM / BUILTIN\Administrators = Full;CASCADES\Ashley.Jensen+CASCADES\Meredith.Kuhn= Modify (no Everyone); share-access limited to the same two + Admins. Mapped persistentE:on DESKTOP-U2DHAP0 (Ashley) and ASSISTMAN-PC (Meredith), RW-verified. NOTE: clients reach CS-SERVER SMB at 192.168.2.248 (registered DNS / Ethernet idx16), NOT the .254 Hyper-V vEthernet NIC. RMM dispatch gotcha: build UNC from[char]92(heredoc+jq eats\\->\); surface a remotely-mapped drive in the user's running Explorer withSHChangeNotify(SHCNE_DRIVEADD)in their session.
Printers / VLAN 20 Migration (2026-06-30)
- Server-share model is preferred (Howard) for domain-joined machines:
\\CS-SERVER\<share>print queue rather than direct-IP. Workgroup (not-yet-domain-joined) machines get a direct-IP local printer installed as SYSTEM (machine-wide/all users) -- no share, no domain auth, no point-and-print prompt -- switch to\\CS-SERVER\<share>once domain-joined. - Point-and-Print policy is the correct "apply our admin rights" fix for standard users pulling a driver from CS-SERVER (NOT making users local admins). Symptom without it: mapping elevation prompt
0x800702e4and per-machine/gasilently failing at logon (PrintService event 513 / error 0xBCB) -- both are the defaultRestrictDriverInstallationToAdministratorsblocking the driver pull. SetHKLM\SOFTWARE\Policies\Microsoft\Windows NT\Printers:RestrictDriverInstallationToAdministrators=0+ subkeyPointAndPrint(Restricted=1, TrustedServers=1, ServerList=CS-SERVER, NoWarningNoElevationOnInstall=1, UpdatePromptSettings=2), scoped to CS-SERVER. Caregiver machines already have this (why their printer GPO works); this is the prerequisite layer a printer-deployment GPO needs fleet-wide. - Canon MF741CDW / MF743CDW are UFR II ONLY -- NOT PCL. A PCL6 driver produces Error #822 (spools but nothing prints). CS-SERVER only staged PCL6/PS3/XPS; pulled
Canon Generic Plus UFR II V250(INFcnlb0ma64.inf) from a client DriverStore to CS-SERVER (client ->\\CS-SERVER\C$, since CS-SERVER off-subnet can't reach a client's C$). - Static VLAN 20 printer IPs (2026-06-30): Front Desk Epson ET-5800 =
10.0.20.221(share\\CS-SERVER\FrontDesk); Life Enrichment Canon MF741CDW =10.0.20.94(share\\CS-SERVER\LifeEnrichment, UFR II; sharon.edwards + susan.hicks, default stays Copy Room); Business Office Brother L8900CDW =10.0.20.220; Dining Room Canon MF743CDW =10.0.20.228(DESKTOP-MD6UQI3, direct-IP UFR II); Chef Office Brother MFC-9330CDW =10.0.20.236(CHEF-PC, direct-IP; JD Martin's default stays USB Chef Printer); MedTech Brother MFC-L8900CDW =10.0.20.74(memcare box + DESKTOP-LPOPV30/Karen); MC Reception Epson ET-5800 =10.0.20.78(MEMREct-PC, not yet set up). Running map:docs/printer-gpo-map.md. - [PLANNED] Printer GPO: put the Point-and-Print policy in a fleet-wide computer GPO; repoint
CSC - Life Enrichment PrintersGPO to\\CS-SERVER\LifeEnrichment(old ref1F-132-RecRoom-Canon); build per-room printer-deployment items as machines domain-join.CSC - Printer Deploymentis the known disabled/empty/reference-only GPO. Howard floated packaging the migration how-to (VLAN routing/pfSense bypass, server-share repoint, Point-and-Print, UFR II, [char]92 UNC) into a reusable Cascades printer skill.
Synology NAS (cascadesDS) / Shared File Access
- Device specs (confirmed live 2026-06-25 via
synologyskill): DS718+, DSM 7.2.1-69057 Update 11, 6 GB RAM, serial 1920PEN537202. Filesystem ext4 (NOT Btrfs); 2x WD10EZEX 1 TB (volume1). 10 shares (homes, Public, SalesDept, Server, Management + hiddenpacs,Activities,chat,Sandra Fish,web). 30 packages running incl. Active Backup for Business 3.1.0, Synology Drive Server 3.5.0, Chat, VPN Server, Hybrid Share. Reachable only with the Cascades site VPN up. - Stale Word owner (lock) files on cascadesDS shares: Word creates a hidden
~$<truncated filename>owner file when a document is opened; orphaned on abrupt session end. Fix: delete the~$file(s). Confirmed 2026-06-10. - Accessing cascadesDS from RMM -- always use a user session, not CS-SERVER SYSTEM. The domain-joined CS-SERVER machine account cannot authenticate to the Synology
Publicshare because cascadesDS uses workgroup "CASCADES" (same short name as the AD domain), causing Kerberos auth failures. Run the command inuser_sessioncontext of a machine where the target user is actively logged in. - Synology Drive sync scope (as of 2026-06-18): The Drive Client on CS-SERVER syncs only the Sync DSM user's My Drive (
/volume1/homes/Sync/Drive/) intoD:\Shares\Main-- one-way download. The real department shared folders are NOT in this scope. Note:synopkg status SynologyDrivefalsely returns "stopped" (status 263) even when active -- verify viasystemctl is-active pkgctl-SynologyDriveandnetstat -tlnp | grep 6690.
CS-SERVER SMB & Endpoint AV (2026-06-26)
- The "CS-SERVER SMB error 67 outage" was a TEST-METHOD ARTIFACT, not a real outage. RMM-dispatched SMB client commands (
net use/net view/Test-Path/Get-SmbConnection, even inuser_session) false-negative -- they return error 67 (BAD_NETWORK_NAME) / RPC 1702 / "none" even for KNOWN-GOOD targets. CS-SERVER SMB is healthy --Get-SmbSessionshowed 7 live SMB 3.1.1 users / 30 open files / new sessions forming. VALIDATE SMB server-side (Get-SmbSession/Get-SmbOpenFile) or with a REAL interactive test -- never from RMM client-side results. A drive-mapverifyfailure is NOT proof of a problem (skill caveat added; errorlogrmm/smb-testing). - CS-SERVER endpoint AV was DattoAV, not GravityZone Bitdefender. It was the Datto EDR "Endpoint Protection SDK" (Bitdefender engine + Avira "Sentry" driver -> drivers
BdSentry/rtp1/rtp2), managed by Datto RMM (CentraStage/CagService) + Datto EDR Agent (HUNTAgent/Infocyte HUNT, tenant azcomp4587). Removing the box from the GravityZone console did nothing because GravityZone never managed it. ALL Datto software was fully removed from CS-SERVER 2026-06-26 (services deleted,infocyte/CentraStagedirs gone, registry + kernel drivers cleared). CS-SERVER was already de-enrolled from the EDR tenant, so no uninstall token could be issued -- forced removal once the tamper drivers were gone. - Karen Rossini share access -- RESOLVED.
CASCADES\karen.rossini(reset + vaultedclients/cascades-tucson/karen-rossini.sops.yaml, member ofSG-IT-RW) verified able to open\CS-SERVER\Servershares interactively from another PC. Her ALDocs desktop shortcut + Quick Access pin were set on DESKTOP-LPOPV30 (\CS-SERVER\Server\ALDocs) via thedrive-mapskill. Note: her earlier move to CSCNet (WPA3-SAE) broke NAS-by-name resolution (unrelated side effect).
Datto EDR / Bitdefender Decommission
- Datto EDR/AV (Infocyte/azcomp4587) is the ACG-managed endpoint stack (org
2d5ea96e, target group1dbd2b02, reg key6qw68y2rwl). Install one-liner via/rmm:Install-EDR -URL "https://azcomp4587.infocyte.com" -RegKey 6qw68y2rwl. - Bitdefender GravityZone BEST removal is CONSOLE-ONLY. The GravityZone Public API
createUninstallTaskis DEAD in this version and BEST anti-tampering (no uninstall password on policy "GPS Default") blocks endpoint-side uninstall. Use the console "Uninstall client" task (Network -> Cascades company66b0448e-> machine -> Tasks). New/replacement machines: provision Datto EDR/AV only, never Bitdefender, and never carry over prior-MSP Datto RMM/CentraStage artifacts. - RECEPTIONIST-PC is two physical boxes sharing a hostname (frontdesk MJ0KQHNP, MemCare MJ0KQH4R -> being renamed MEMCARE-STATION). Reconcile fleets by SERIAL, not hostname -- dedup-by-hostname masks the second box's coverage gaps.
- BD detection script: services
^EP(Security|Protected|Update|Redline|Integration)Service$+ uninstall-registry DisplayName ~Bitdefender|GravityZone+Test-Path 'C:\Program Files\Bitdefender'.
Browser / Edge
- [BUG - FLEET] Edge 149 cannot open Office files via download-list when Downloads is a UNC-redirected folder (Chromium issue 519243472). A regression introduced in Chromium 149 prepends
\\?\to UNC paths without converting to the correct\\?\UNC\form. Symptom: clicking.xlsxor.docxin the Edge download panel shows "Windows cannot find '\?\\cs-server...'". Text files and PDFs open fine. Trigger: Downloads folder redirected via GPO Folder Redirection to a UNC path. Affected build: Edge stable 149.0.4022.52. Fix options (none applied as of 2026-06-08): (1) Update Edge past the fix; (2) Interim:--disable-features=LaunchShellExecuteViaExplorer; (3) Zero-config: use "Show in folder" then double-click from Explorer; (4) Rollback to 148. Note: pinning to 148 forfeits security fixes; prefer option 1 or 3 for HIPAA machines.
Conditional Access / Caregiver Policies
- Phased rollout -- never tenant-wide. CA policies for caregivers now target
SG-Caregivers(8b8d9222-5d71-419a-936d-56d895c6c332). The legacy "Require MFA for all users" policy stays in place. All 40 real caregivers are now inSG-Caregivers+ Business Premium licensed (2026-06-30). - Enforced caregiver CA policy set (unchanged as of 2026-06-03):
CSC - Block caregivers off Cascades network(e35614e1-e896-4a13-9407-076963af488f) -- BLOCK if location not CascadesCSC - Block caregivers on non-compliant device(ede985e2-ee7e-4521-88b2-34c847c3db20) -- BLOCK if device non-compliant. Pending DISABLE at allow-list cutover.CSC - Caregiver sign-in frequency 8h(7d491c7a-ad90-4420-9990-40a1e676a76c)
- Caregiver device allow-list (2026-06-03 -- report-only):
CSC - Caregivers: allow-listed devices only (REPORT-ONLY)-- id1b7fd025-1aad-47c8-9274-c32c3e0b163c; stateenabledForReportingButNotEnforced. Device filter (modeexclude):(device.displayName -startsWith "CSC-") -or (device.extensionAttribute1 -eq "CSCCaregiverDevice"). Includes: NURSESTATION-PC (deviceIdd3bf931f), Laptop2, LAPTOP-DRQ5L558, LAPTOP-E0STJJE8, LAPTOP-8P7HDSEI, ASSISTNURSE-PC (needs re-join + re-tag after Win11 reinstall). - GDAP exclusion: CA policy 3 must exclude "Service provider users" (GDAP foreign principals) +
SG-External-Signin-Allowed+SG-Break-Glass, otherwise ACG partner admins lose access at CA cutover. - Known bug:
Require MFA for all userspolicy (7e87a1c7...) excludesSG-Caregivers-Pilotinstead of the liveSG-Caregivers(8b8d9222). Functionally harmless today (pilot group still exists), but must be corrected. - Pilot cleanup required when done: Delete
pilot.test@cascadestucson.com, clean uphoward.enos@cascadestucson.com, removeSG-Caregivers-Pilotfrom CA policy targets and delete the group.
EXO / Message Trace
- Get-MessageTrace is deprecated. Use
Get-MessageTraceV2instead. V2 has a 10-day max window -- loop 9 consecutive windows to cover 90 days. - EXO access token auth: When
Connect-ExchangeOnline -Credentialfails and the app cert is not in the Windows cert store, use client_credentials flow to get an EXO-scoped token and pass it via-AccessToken.
Wireless / UniFi RF
- [APPLIED 2026-06-19 -- validated] Production RF optimization applied + kept:
- 2.4 power -> MEDIUM on 47 radios (42 over-thinned-
lowfloors 1-4 + 5 MemCareauto/full radios 505/517/608/615/622). The 24 thinned-disabled radios stayed disabled; 5 mesh-auto APs untouched. Non-regressive. Per-AP targeting required --apply-radio power --zonere-enables disabled radios (confirmed gotcha). - 5 GHz -> clean DFS 40 MHz channels on 72 non-mesh APs (channels 52/60/100/108/116/124/132/140), 0 co-channel, mesh excluded. Result: 5 GHz retry roughly HALVED -- 8.7 -> 3.8 avg, median 8.2 -> 2.1. Validated; all 72 APs holding DFS, 0 radar vacates. Voice nudged back to 5 GHz (kick-sta).
- CSCNet BSS-transition (802.11v) ON. 6 GHz BLOCKED (WPA3 mandate on WPA2/PPSK SSID -- deferred to Howard).
- [KEY DATA -- DFS decision reversed] Full channel survey (74/74 APs) proved DFS channels here are 4-5x cleaner (2-3% busy) than non-DFS (ch149=12%, ch157=28%, ch44=22% -- the property's worst channels). Consumer/neighbor gear avoids DFS; choosing channels from measured scan (not a non-DFS policy) delivered the win. Always: scan ->
survey-report.py->channel-plan --channels-> apply -> validate. Do NOT use UniFi auto-channel; do NOT apply a non-DFS-only policy without checking the survey data for this site.
- 2.4 power -> MEDIUM on 47 radios (42 over-thinned-
- Fleet (full audit 2026-06-16): 77 U7-Pro APs, 12 switches, ~587 wireless clients. Controller: UOS at 172.16.3.29, HTTPS 11443; site short name
va6iba3v. No UniFi gateway (pfSense is the gateway). - Primary pain band is 2.4 GHz. Avg TX-retry ~10%; cu_total 69-94% live; catastrophic external neighbor BSSID density (ch6 ~33k BSSIDs, ch1 ~19k, ch11 ~17k). 27 of the 40 worst clients stuck on 2.4 GHz, mostly IoT/legacy hardware. Experience splits by band: 5/6 GHz clients are fine; clients that land or stick on 2.4 GHz suffer.
- 6 GHz is nearly unused -- root cause: CSCNet not broadcasting 6 GHz (
wlan_bands=[2g,5g], found 2026-06-18). 75 radios active but only ~1 client because the band is dark at the SSID level. Largest untapped, clean, non-DFS capacity. Enabling requires WPA3+PMF conversion on the 427-client SSID -- Howard's supervised decision. - Poly phone drops (2026-06-17) -- CLOSED. Root cause = intentional pfSense reboot on 2026-06-16 22:38:12 MST (one fleet-wide event; 28/30 phones each dropped once). Only a gateway-level event explains all-floors-at-once.
- DHCP is healthy. pfSense dhcpd.log: 1241 ACK / 1 NAK / 0 no-free-leases. Per-room /28 HIPAA segmentation is intentional; do NOT flatten.
sta_dhcp_failuresmetric is client/WiFi-side, not pfSense-side. - Switch audit (2026-06-16): ~25 ports linked at 100 Mbps but gig-capable (systematic cabling/NIC issue, 1st/2nd/3rd-floor switches; investigate after WiFi Phase A). 3 offline switches: Switch 2nd Floor #2 (reset+re-adopted 2026-06-17), Switch 4th Floor #2, USW Pro Max 16. Port p38 (1st Floor USW) 4.0% tx-drop rate.
- Mesh topology: 2nd Floor Atrium is wireless-mesh parent for CC Bridge + salon (5 GHz backhaul ch36); 206 U7 Pro carries AP 108. These must NEVER be disabled or powered down via zone command -- coverage-thin auto-excludes them.
- AP-hang recovery: use
device-control.sh cascades poe-cycle "<AP name>" --apply. Do NOT useforce-provision-- it took AP 445 offline during the Floor-4 pilot. - Tooling (
unifi-wifiskill -- feature-complete as of 2026-06-19):- Collectors:
audit-site.sh,live-stats.sh,model-rank.sh,radio-usage.sh,coverage-thin.sh,neighbor-collect.sh,survey-collect.sh,dfs-check.sh,switch-audit.sh,gw-audit.sh,monitor-run.sh,sites.sh. survey-report.py(NEW 2026-06-19) -- the channel-decision driver: rollssurvey-collectJSON into the fleet per-channel/per-band-group measured busy% table + cleanest/dirtiest ranking + suggested clean 40 MHz palette. Run it BEFORE any channel change; it's what makes the DFS-vs-non-DFS call from facts. Previouslysurvey-collect's report ANDchannel-plan's palette had a non-DFS bias baked in -- both fixed 2026-06-19.- Apply (gated + rollback):
apply-radio.sh,apply-wlan.sh,client-control.sh,device-control.sh,channel-plan.sh(DATA-DRIVEN:--channels <list>or--dfs ok|avoid|only). - pfSense:
pfsense-ssh.sh(audit/dhcp/run -- SSH backend, no RESTAPI package needed; blocked over VPN -- onsite only).
- Collectors:
- Creds (vault refs only):
infrastructure/uos-server-ssh-key(SSH/Mongo),infrastructure/uos-server-network-api-rw(RW API),clients/cascades-tucson/unifi-ap-ssh(per-AP SSH, needs site VPN),clients/cascades-tucson/pfsense-firewall(pfSense admin for pfsense-ssh.sh).
VoIP / Network Device Migration
- Re-VLANing a wired switch port requires a link bounce to force re-DHCP. Changing the native VLAN on a UniFi switch port does not reset the NIC link; the device holds its old DHCP lease. Fix: bounce the port (PoE power-cycle for PoE devices; disable/enable via controller API for non-PoE). A UniFi client block/unblock is a MAC-address filter only -- it does NOT bounce the link. Controller API port-bounce requires the
X-CSRF-Tokenfrom the login response header (x-updated-csrf-token). - Externally-powered devices (AudioCodes desk phones) need a PHYSICAL power-cycle, not a controller bounce. The 8 AudioCodes sit on USW-16-PoE ports 1-8 but run on external power bricks (PoE OFF on those ports) -- so a UniFi PoE power-cycle AND a controller port disable/enable are both no-ops. They held their old main-LAN DHCP leases until Howard physically powered each off/on (2026-06-18) -> VOICE leases
.224-.231. - UniFi controller PUT 403 / CSRF: rapid controller writes can 403 -- read the CSRF token from the
x-updated-csrf-tokenresponse header (TOKEN-cookie JWT as fallback). - API scratch files must be written OUTSIDE the repo. Controller-scratch written CWD-relative got swept into commits. Use
mktemp -doutside the repo;.gitignorepatterns (.fleet*,.ap[0-9]*,.vq[0-9]*,.q[0-9]*) added as a backstop. - Verify VLAN membership via the client
vlanfield, not the controller's displayed IP. IP field caches/lags (Kitchen server phone showed stale 192.168.1.126 while actually on vlan:30). - Build UNC from
[char]92in RMM PowerShell payloads. Literal\\collapses to a single\through jq/agent, mangling the path. Recurs across printer + share dispatch.
Voice QoS (VLAN 30) -- design (2026-06-18, NOT yet built)
Full design: docs/network/phase1-voice-qos-design.md. Status DESIGN -- nothing applied.
- The VLAN move's QoS payoff: all voice is one subnet
10.0.30.0/24, so QoS matches all voice by source subnet -- no per-PBX SIP/RTP port guessing. Phones confirmed marking DSCP EF (46). - QoS is INSURANCE, not the everyday fix. Measured WAN1 fiber upload ~522 Mbps vs ~98 Mbps peak usage -> WAN is not the day-to-day constraint. QoS earns its place for (1) WAN2 (coax) failover and (2) rare WAN1 saturation. The everyday dropped-calls cause is RF (band selection).
- Layered design: L1 pfSense HFSC shaper on BOTH WANs -- 3 queues
qVoice(prio 7, realtime ~30%, source10.0.30.0/24via floating out rule),qACK(~10%),qDefault(~60%). L2 UniFi WMM maps DSCP EF -> WiFi Voice AC. L3 UniFi switch QoS. Blocker for L1 sizing: WAN2 coax upload number. - Build path: pfSense GUI -> Traffic Shaper -> Wizard "Multiple Lan/Wan". Howard drives. Rollback = disable/remove the shaper (zero residual; reverts to FIFO).
pfSense Operations
- pfSense 25.07 logs are PLAIN TEXT, not binary clog. Read with
tail/grepdirectly. Usingclogreturns empty output and will cause false conclusions. - [2026-06-30] LAN "allow LAN to any" rule policy-routes internal-VLAN traffic out the WAN. The rule's Gateway = WAN_Group (dual-WAN), so CS-SERVER -> VLAN 20 (and any LAN -> internal-VLAN) traffic gets shoved out a WAN and fails -- NOT a firewall block. Fix: a top LAN pass rule, Gateway = default, scoped to the source (e.g. src CS-SERVER
192.168.2.248, dst10.0.20.0/24) so residents/guests on other ingress interfaces don't match. Do it in the GUI (SSH/22 is blocked from the OpenVPN subnet). - pfSense OpenVPN
--inactiveidle timeout: ~300s; disconnects idle clients after ~5 min of no tunnel data. Keepalive pings do NOT reset it. Fix: raise/disable--inactiveon the server profile. Proposed 2026-06-18; not applied. - pfSense SSH/22 is blocked from the OpenVPN subnet (GUI 443 open). The unifi-wifi
pfsense-ssh.shskill silently returns empty over VPN (sends ssh stderr to /dev/null) -- use the GUI remotely, or SSH from onsite. - pfSense dirty-boot / duplicate dhcpd: After an unclean pfSense shutdown, dhcpd may start twice. Fix:
killall dhcpd && echo "services_dhcpd_configure();" | /usr/local/sbin/pfSsh.php; verify one instance:pgrep -f "dhcpd -user" | wc -l== 1. Note:pfSsh.phpis slow (~20-40s); use timeout 60s+. - Post-outage device stragglers: Devices that booted during a DHCP-down window cache a disconnected state and do not retry once the network recovers. Realistic plan: reactive power-cycle as reports come in. Cox modem must be rebooted after a pfSense configuration restore.
Known Issues / Pending Hygiene (as of 2026-06-30)
- [BUG] Stale exclude-group on MFA-all-users policy: The
Require multifactor authentication for all userspolicy (7e87a1c7...) excludesSG-Caregivers-Pilot(0674f0bc...) instead of the liveSG-Caregivers(8b8d9222...). Fix: PATCHexcludeGroups. - [DESIGN] ALIS-native 2FA is not a perimeter control. Force all ALIS logins through Entra SSO (SSO-only, credential fallback disabled); disable ALIS-native 2FA per-user then globally.
- [INFO] Android enrollment token expiry (2027-05-08) does NOT unenroll devices. Renewal needed only before enrolling new devices after that date.
- [WARN] ~25 switch ports at 100 Mbps but gig-capable. Investigate after WiFi optimization is stable.
- [WARN] 3 offline switches (Switch 4th Floor #2, USW Pro Max 16 -- root cause unknown; Switch 2nd Floor #2 was reset+re-adopted 2026-06-17). Investigate onsite.
- [SECURITY] Synology Cloud Signin Portal credential exposed in vault git history (commit 1fbc0e1). Encrypted go-forward but credential must be rotated.
- [FLEET] Endpoint security migration substantially complete (2026-06-30). Datto EDR/AV (Infocyte/azcomp4587) is the ACG-managed endpoint stack -- ~34 agents enrolled. Bitdefender removed from RECEPTIONIST-PC (both boxes); orphaned BD folders cleaned; CS-SERVER fully Datto-free. Pending: DESKTOP-TRCIEJA BD_ACTIVE (needs GravityZone console uninstall; box slated for replacement); DESKTOP-F94M8UT (Alma's, powered off) queued EDR install; confirm Cascades removed from Syncro's BD deployment; GravityZone portal cleanup of stale RECEPTIONIST-PC records; reconcile laptop3 (EDR, no RMM agent); confirm/remove stale EDR agents laptop1 + cascades-laptop.
Security Incidents (historical)
-
Megan Hiatt (2026-04-16): Active credential-stuffing -- 126 failed sign-ins, bursts from Belfast GB, Hamburg DE. Password reset and SMTP AUTH disable were action items. Mailbox was clean (not breached). Re-verify the April remediation held (flagged again during Tamra offboarding 2026-06-30).
-
John Trozzi (2026-04-16, 2026-04-20): Investigated twice -- both times NO BREACH. First: credential stuffing flag (clean). Second: inbound phishing email (clean).
-
Crystal Rodriguez (2026-04-19): Phishing investigation. Report:
clients/cascades-tucson/reports/2026-04-19-crystal-rodriguez-phish-investigation.md. -
Canva email delivery (2026-05-20): Alma Montt not receiving Canva invites. Resolved by adding canva.com domains to AllowedSenderDomains in EOP policies.
-
ALIS AADSTS65001 (2026-06-03): megan.hiatt, karen.rossini, memcarereceptionist could not sign in to ALIS on non-phone devices. Root cause: missing tenant-wide admin consent on ALIS SP (
e1cae4ad). Resolved by grantingAllPrincipalsUser.Readvia Graph API. -
dunedolly21@gmail.com: External guest invited 2026-04-14 by Lauren Hasselman from mobile. Status unknown -- confirm with Lauren. [unverified]
-
Chris Knight bill.com / BOK email delivery (2026-06-04): Root cause was SENDER-SIDE: bill.com address on SendGrid suppression list; BOK had wrong recipient email. Resolved externally by Howard. No tenant config changes needed. Ticket #32383, Resolved.
HIPAA Compliance
- Primary objective. Cascades stores PHI on CS-SERVER and uses ALIS for clinical records.
- Critical open gaps: No audit logging on D:\Homes (SS164.312(b)); Object Access auditing disabled; no SMB encryption on homes share. Audit retention infra (LAW 90d + Storage 6yr) approved but not yet built.
- Backup gap closed (2026-06-15): Mike installed ACG cloud backup (MSP360/CloudBerry) on CS-SERVER. Verify first full completes + confirm image-based / bare-metal + system-state + retention before any drive work.
- Restored 7 deleted mailboxes (2026-04-25) for HIPAA SS164.316(b)(2) 7-year retention.
- Termination policy established: Convert to shared mailbox, hide from GAL, retain 7 years. Litigation Hold is available (Exchange Plan 2) but has been waived for the two 2026-06 offboardings (Alma, Tamra) per Howard's explicit no-PHI/PHI-adjacent-but-authorized calls.
- Voice VLAN 30 (HIPAA-isolated): All voice gear on an isolated network with internet/cloud-PBX egress only; blocked from PHI/LAN/VLAN20/mgmt. Migration COMPLETE 2026-06-19: 37 devices on VOICE (28 Poly + 8 AudioCodes + desktop).
Active Work
Canonical remaining-work plan:
docs/REMAINING-WORK-PLAN.md(built 2026-06-24 from a live AD+RMM domain-join diff). 7 sequenced workstreams + every open ticket mapped to one. Work from it.
Syncro live pull 2026-06-30: 0 open Syncro tickets; 37.5 prepaid hours; 29 managed devices. See session logs for active project work (domain migration, EDR rollout, caregiver phone SSO, VLAN 20 printer migration).
- [IN PROGRESS 2026-06-30] Caregiver phone SSO -- ALIS email-match (Howard handling). Entra/identity side DONE (all 40 caregivers in
SG-Caregivers, Business Premium licensed, forced-change AD temp passwords vaultedclients/cascades-tucson/caregiver-temp-passwords-2026-06-30.sops.yaml). Remaining: set each caregiver's ALIS staffEmail= Entra UPN (23 confirmed just need Email=UPN; 5 blank-role confirm+match; 5 Med Techs revisit; 7 need an ALIS record created; 3 ALIS-only caregivers need AD accounts -- Judith Palmer, Joey Ty, Alejandra Vallejo). Blanket-disable ALIS-native 2FA for the bucket as matched. Zeke Huerta stayse.huerta@. Also decide reactivate-vs-recreate for the 7 Discharged ALIS records (from the 6/29 crosscheck). Build path:alisskillbuild-import-> upload .xls in ALIS UI. - [IN PROGRESS] VLAN 20 (CSCNET) staff + printer migration. Front Desk Epson (.221), Life Enrichment Canon MF741 (.94, UFR II), Dining Canon MF743 (.228), Chef Brother 9330 (.236), MedTech Brother L8900 (.74) done 2026-06-30; MC Reception Epson (.78) marked but not set up. pfSense CS-SERVER->VLAN20 policy-route bypass rule added. Next: Point-and-Print into a fleet-wide computer GPO; repoint
CSC - Life Enrichment PrintersGPO to\\CS-SERVER\LifeEnrichment; reboot MEMCARE-STATION to apply rename; domain-join the workgroup boxes (DESKTOP-MD6UQI3, CHEF-PC, MEMCARE-STATION, MEMREct-PC, DESKTOP-LPOPV30) then swap direct-IP printers to server shares. Map:docs/printer-gpo-map.md. - [SECURITY -- needs Global Admin] Remove the standing Privileged Authentication Administrator role from the
ComputerGuru - Tenant AdminSP (left over from Alma's offboarding password reset). Entra -> Roles & admins -> Privileged Authentication Administrator -> remove the SP; leave its Conditional Access Administrator role. Pending Mike's decision. See Access section. - [FOLLOW-UP 2026-06-30] Megan Hiatt breach re-check. Her account carried a
CREDENTIAL_STUFFING_ACTIVEmarker in the April tenant inventory; verify the April remediation held (/remediation-tool check megan.hiatt@). - [PLANNED -- CARF accreditation] Technology and System Plan deliverable (requested by Ashley Jensen 2026-06-24). One of the five required CARF Section-1 plans (Aging Services); must be an action document covering 8 canonical areas with per-area current tech + projected need + timeline + vendor + cost + responsible person + target/completion date, annual dated leadership sign-off. Done: gap analysis, project memory
project_cascades_carf_tech_plan, an on-brand PDF first pass (viaimpeccable), and a pre-filled CARF intake worksheet. Next: gather Cascades' inputs, then build the final plan branded as Cascades' (ACG as preparer); confirm the exact standard citation + review cadence. Standing rule: all client/vendor-facing deliverables run through theimpeccableskill before delivery. - [URGENT] Order replacement workstation for Lupe Sanchez (DESKTOP-TRCIEJA). Decision made 2026-06-18. EOL Gateway ZX6971 / i3-2120 / 8 GB / Win11-unsupported. On new machine: provision GuruRMM + Datto EDR/AV only; do NOT install Bitdefender. Do not carry over any prior-MSP Datto RMM/CentraStage artifacts. (TRCIEJA is currently BD_ACTIVE -- needs GravityZone console uninstall if kept in service before replacement.)
- [IN PROGRESS] Datto EDR/AV rollout + Bitdefender decommission. ~34 agents enrolled (org
2d5ea96e). Remaining: DESKTOP-TRCIEJA GravityZone console uninstall (BD_ACTIVE); DESKTOP-F94M8UT (Alma's, powered off) queued BD-aware EDR install (cmda4623704); (1) Remove Cascades from Syncro's Bitdefender deployment (GUI-only). (2) GravityZone portal cleanup of stale RECEPTIONIST-PC endpoint records (company66b0448e). (3) Reconcile laptop3 (EDR active, no RMM agent). (4) Confirm/remove stale EDR agents laptop1 + cascades-laptop. CS-SERVER Datto stack already fully removed 2026-06-26. - [URGENT] Rotate exposed Synology Cloud Signin Portal credential. Vault commit 1fbc0e1 committed it plaintext; encrypted go-forward but exposed in git history. Also verify MDM service account + WiFi CSCNet from that commit were never plaintext.
- [DONE 2026-06-19] Voice VLAN (VLAN 30) migration COMPLETE -- 37 devices on VOICE. All Poly re-keyed. RF optimized (2.4 power->medium, 5 GHz clean DFS, retry halved). Billed ticket #32444 (7h prepaid).
- [PENDING - hardware] Bistro phone replacement. Kitchen server phone was bad (John pulled it 2026-06-19); Bistro phone relocated to Kitchen, so the Bistro has no phone. Set up + re-key the replacement to the voice PPSK when it arrives.
- [WAITING ON VERTICAL - the last voice item] Set Poly handsets to 5 GHz-only. Residual dropped-calls are a band-selection problem: phones on saturated 2.4 GHz despite strong 5 GHz signal; band-steering won't hold the Poly fleet on 5 GHz. Phone-side 5 GHz lock is the fix (or the CSC ENT device-island plan) -- request sent to Richard Turner 2026-06-19 (
docs/network/2026-06-19-vertical-5ghz-lock-request.md), awaiting response. - [INVESTIGATE] Phone
.210-- on 5 GHz at -65 dBm but ~64% retry on a clean channel; anomalous. - [PENDING - build] Voice QoS for VLAN 30 (pfSense HFSC 3-queue on both WANs matching
10.0.30.0/24+ UniFi WMM/switch QoS). Design done, not built. Blocker for sizing: WAN2 coax upload number. - [PENDING - deferred] Enable 6 GHz on CSCNet. Blocked on
Wpa3MandatoryFor6GHzBand-- converting CSCNet from WPA2/PPSK to WPA3+PMF touches all 427 clients. Howard's supervised decision + coordinated change window. - [PENDING] Measure WAN2 (coax) upload -- remote source-route test failed; get from a WAN2-routed host or the Cox bill.
- [PENDING] Re-enable 3 AM AP auto-upgrade (left OFF after 2026-06-19 overnight run).
- [PENDING] Stand up recurring
dfs-check.shradar monitor on the DFS channels (fold into network-logging plan). - [PENDING - next week] MemCare min-RSSI (floors 5/6) -- deferred until Howard adds new APs to floors 5/6.
- [PLANNED] Network logging / observability -- Synology cascadesDS (DSM Log Center syslog server) as collector; pfSense + UniFi + AP syslog sources;
/stat/staclient snapshotter. Spec:docs/network/network-logging-plan.md. Log Center package not yet confirmed installed. - [PENDING] Synology Drive Team Folder migration (department shares -> CS-SERVER). Current Drive sync covers only the Sync-user's My Drive. Pilot on
/volume1/Server(1.9 G) first. - [PENDING] Watch for post-outage device stragglers. Fix each by power-cycle.
- [PENDING] pfSense OpenVPN
--inactivetimeout fix. Raise/disable the ~300s idle timeout on the Cascades OpenVPN server profile. - [PENDING] Enable Netgate AutoConfigBackup on pfSense. Also verify UPS covers all core infra + PoE switches on battery-backed outlets (pfSense rectified; others not confirmed).
- [PLANNED] KPI dashboard (Ashley Jensen): scoped 2026-06-17; client one-pager drafted. Parked pending Ashley's day-one KPIs, data-freshness need, POS/Focus-HR specifics. Confirm ALIS analytics availability with Medtelligent.
- [HOME->PRO upgrades] NurseAssist + DESKTOP-MD6UQI3 upgraded to Pro for Workstations 2026-06-26 (2 x $99 = $198, not yet invoiced -- Syncro product 23571919). SALES4-PC upgraded by its supplier (no ACG charge). CascadesProxess deferred to a late-night window (Proxess access-control server, in active use for the access-control hardware install -- reboot briefly interrupts door access; verify ProxessIQ Server restarts).
Device-readiness for domain migration (2026-06-24 audit + 6/26 upgrades):
- READY to join (Pro/Enterprise, internal): DESKTOP-LPOPV30 (Karen), MAINTENANCE-PC (Bruce), LAPTOP-E0STJJE8; after a reboot: ASSISTMAN-PC (Meredith), ANN-PC, Laptop2; CHEF-PC after #32254. NurseAssist + DESKTOP-MD6UQI3 now Pro (6/26).
- BLOCKED -- Windows Home (until Pro): LAPTOP-8P7HDSEI, MDIRECTOR-PC (Shelby), MEMREct-PC. CascadesProxess deferred (appliance, late-night).
- OneDrive KFM ON (unlink before folder-redirect): LAPTOP-8P7HDSEI, NurseAssist.
- LAPTOP-DRQ5L558 is off the Cascades LAN -- get on-site before join.
- Decision 2026-06-24: caregivers stay TEST-scoped -- do NOT flip the lockdown to go-live until all devices are domain-ready first.
Migration phase status (as of 2026-06-30):
| Machine / User | Status |
|---|---|
| Sharon Edwards (DESKTOP-DLTAGOI) | Domain-joined, folder redirect working via registry workaround. LifeEnrichment printer (UFR II) mapped 6/30. |
| Ashley Jensen (DESKTOP-U2DHAP0) | Domain-joined, folder redirect manually fixed |
| Crystal Rodriguez (CRYSTAL-PC) | Domain-joined, folder redirect confirmed working 2026-05-21 |
| RECEPTIONIST-PC (frontdesk box, MJ0KQHNP) | Domain-joined 2026-05-22; FrontDesk printer (.221) mapped 6/30 |
| RECEPTIONIST-PC (MemCare box, MJ0KQH4R) | Rename to MEMCARE-STATION STAGED 6/30 (applies next reboot); MedTech printer (.74) set up |
| NURSESTATION-PC | Domain-joined, folder redirect complete |
| Lauren Hasselman | Domain-joined, folder redirect complete 2026-05-23 |
| Megan Hiatt (Marketing) | COMPLETE 2026-05-27 |
| DESKTOP-KQSL232 (Lois Lane -- old box) | Decommissioned (removed from EDR straggler list 6/26) |
| HEALTH-SERVICES (Lois Lane) | Domain-joined (confirmed 2026-06-24) |
| CHEF-PC (JD Martin), MDIRECTOR-PC, MEMREct-PC, LAPTOP-8P7HDSEI | Workgroup / Home-blocked -- direct-IP printers now; domain-join + GPO migration pending |
| NurseAssist, DESKTOP-MD6UQI3 (Dining/Alyssa) | Upgraded to Pro 6/26; DESKTOP-MD6UQI3 got direct-IP Dining printer (.228). Domain-join pending. |
| SALES4-PC (Tamra -- OFFBOARDED) | User offboarded 6/30; machine upgraded to Pro by supplier |
| ASSISTMAN-PC (Meredith), ANN-PC, DESKTOP-LPOPV30 (Karen), MAINTENANCE-PC (Bruce) | Pro/Enterprise + internal -- READY to join |
| DESKTOP-TRCIEJA (Lupe Sanchez) | EOL hardware -- replace instead of migrate. BD_ACTIVE (needs console uninstall if kept). |
Blocking issues / pending:
- M365 relicensing: remaining suspended Business Standard users -> Business Premium (O365 Standard SKU SUSPENDED)
- Break-glass accounts: not created (confirmed 2026-05-27); YubiKey arrival unconfirmed
- Audit retention infra: approved 2026-04-29, not yet built
- Entra Connect: OU=Administrative not yet in sync scope; UPN suffix updates for that OU pending
- NURSESTATION-PC: reboot required to activate
CSC - Caregiver Device LockdownGPO (deployed 2026-06-05) - Caregiver device allow-list: ASSISTNURSE-PC needs re-join + re-tag after Win11 reinstall; LAPTOP-8P7HDSEI Win11 upgrade + join/tag pending; then cutover
- ALIS office/privileged standardization: move office/managers/nurses to ALIS SSO-only; disable ALIS-native 2FA per-user then globally
- Fix stale
SG-Caregivers-Pilotexclude-group onRequire MFA for all userspolicy - ALIS app session timeout: lower from 20 to 15 min (Howard, ALIS admin) -- PENDING
- [CORRECTED 2026-06-24] CS-SERVER RAID is HEALTHY (live OMSA), not degraded. No emergency drive swap. Planned reliability upgrade: replace the 2 consumer 320 GB drives with the 2x enterprise SSD already purchased, on a scheduled window. [WARN] PSU redundancy lost. Service Tag 9MQFTK1.
- [INFO] CS-SERVER cloud backup (MSP360/CloudBerry): verified running 2026-06-24. Still confirm image-based/bare-metal/system-state + retention.
History Highlights
| Date | Event |
|---|---|
| 2026-03-06 | ACG onboarding begins. Initial audit (CS-SERVER Dell R610, pfSense, UniFi, Synology). 19 machines. No backup, no HIPAA compliance. |
| 2026-03-09 | AD security hardening: Monica Ramirez removed from Domain Admins, lockout policy fixed, AD Recycle Bin enabled, MachineAccountQuota set to 0. |
| 2026-03-31 | Cascades onboarded to remediation tool. Tenant ID documented. 50 users, Secure Score 34%. |
| 2026-04-13 | Major onsite: 13 stale AD accounts deleted, OU structure cleaned, UPNs migrated to cascadestucson.com, Homes share created, Folder Redirection GPO deployed (registry workaround), first domain joins. |
| 2026-04-14 | Sandra Fish global admin revoked. ALIS SSO confirmed. Business Premium proposal created. |
| 2026-04-16 | Breach checks: Megan Hiatt (credential stuffing, not breached; password reset). John Trozzi (clean). Crystal Rodriguez phish. /remediation-tool skill built. |
| 2026-04-17 | Howard onsite: folder redirect Sharon Edwards diagnosis. John Trozzi WiFi (TP-Link + UniFi roaming instability). |
| 2026-04-25 | Entra Connect installed on CS-SERVER (staging mode). 7 deleted mailboxes restored for HIPAA. Dual-WAN discovered. |
| 2026-04-28-29 | CA policy reconciliation. Audit retention architecture (ACG-billed, LAW 90d + Storage 6yr). Break-glass design (2 accounts, YubiKeys). Caregiver pilot scope corrected (phased only). |
| 2026-04-30 | CA rollout (Report-only mode): 3 caregiver policies created. SDM bootstrap. |
| 2026-05-01 | Howard billed 33.5 hrs against prepaid block on Entra project ticket #32214 ($0 invoice). |
| 2026-05-07-08 | SDM phone provisioning. SDM token success. ALIS SSO app registration values captured to vault. |
| 2026-05-14-16 | Caregiver AD accounts created. Entra Connect exited staging -- actively syncing. Wireless diagnostic (read-only via cloud API; 2.4 GHz saturation hypothesis identified). |
| 2026-05-18 | Billing review. 39.5 hrs remaining before session. 7 hrs billed separately. |
| 2026-05-20 | Canva email delivery resolved (canva.com domains added to EOP). |
| 2026-05-21 | Crystal Rodriguez folder redirect confirmed working. |
| 2026-05-22 | Ashley Jensen domain-joined. RECEPTIONIST-PC domain-joined. GPO ILT fixes. cascadesDS auth failure diagnosed (workgroup collision) and deferred. |
| 2026-05-23 | Lauren Hasselman folder redirect complete. Megan Hiatt (Marketing) confirmed in AD, domain join pending. |
| 2026-05-26 | Access control vendor meeting onsite (ticket #32324). 0.5h Howard + 0.5h Mike billed. Block at 28.0h. |
| 2026-06-03 | ALIS AADSTS65001 diagnosed and resolved: granted tenant-wide admin consent on ALIS SP e1cae4ad. Caregiver device allow-list CA policy created in report-only (1b7fd025). |
| 2026-06-04 | Three same-day tickets: #32381 Tamra scanner (0.5h onsite), #32382 Megan file access (1.5h onsite), #32383 Chris Knight bill.com/BOK email delivery (1.5h remote). Root cause sender-side. |
| 2026-06-05 | NURSESTATION-PC localadmin login-screen issue resolved. Caregiver test rig built. Hybrid Entra Join + GPOs deployed: CSC - Caregiver Workstation validated; CSC - Caregiver Device Lockdown deployed to OU=Caregiver Devices. Ticket #32303 billed 7.0h, invoice #67782 ($0.00 prepaid). |
| 2026-06-08 | Chris Knight workstation setup (onsite). DESKTOP-N5G1ROO domain-joined + GuruRMM-enrolled. MAJOR: root-caused native Folder Redirection failure -- FR GPO targets were in misnamed fdeploy1.ini; fixed by writing correct fdeploy.ini + version bump. ASSISTNURSE-PC reinstalled (Win10->Win11). Edge UNC download bug diagnosed (no fix applied). |
| 2026-06-09 | Accounting scan-to-folder built. D:\Shares\Accounting on CS-SERVER; shared as \\CS-SERVER\AcctDept; svc-scan service account vaulted; Brother MFC-L8900CDW Scan-to-Network configured (NTLMv2, confirmed). Persistent drive maps set. |
| 2026-06-10 | Meredith Kuhn locked Word doc -- stale owner files on cascadesDS. Five orphaned ~$ files deleted via RMM. Ticket #32403, 0.5h remote, block 56.75->56.25. |
| 2026-06-12 | Created shared mailboxes grievances@ + Surveys@ and delegated to Meredith & Ashley. All 8 permission grants verified. Ticket #32417, 0.5h remote, block 56.25->55.75. |
| 2026-06-15 | Wireless RF full audit -- controller access gained. Mike vaulted SSH key + RW admin + AP SSH. Live audit confirmed 77 U7-Pro APs, ~574->587 clients, 2.4 GHz saturation as primary pain band. |
| 2026-06-15 | CS-SERVER slowness root-caused to (then) degraded RAID-1; cloud backup started; pfSense OpenVPN password reset. MSP360/CloudBerry cloud backup installed on CS-SERVER (closes HIPAA backup gap). |
| 2026-06-16 | Voice VLAN plan for Vertical phones (PLANNED). Designed VLAN 30 VOICE; cutover runbook written. Floor-4 2.4 GHz power-down pilot applied. dfs-check.sh confirmed ZERO real radar events fleet-wide. unifi-wifi skill feature-complete. |
| 2026-06-16 | pfSense confirmed as pfSense Plus 25.07-RELEASE; health verified; Howard-Home LAN renumbered (removed collision with Cascades). pfsense-ssh.sh built. |
| 2026-06-17 | Voice VLAN 30 built + verified; Vertical desktop + initial Poly phones migrated. pfSense igc1.30 interface + isolation rules built. |
| 2026-06-17 | Power outage -- full site down + recovery. pfSense on UPS surge-only side -> unclean shutdown -> duplicate dhcpd + 2nd-floor switch one-way L2. Recovered; pfSense moved to battery, config vaulted. Pre-existing plaintext Synology signin credential found (vault commit 1fbc0e1). |
| 2026-06-17 | KPI dashboard scoping (advisory). 9 reporting systems catalogued. Recommended Phase 1 (exports->SharePoint->Power BI Pro). Parked. |
| 2026-06-18 | Voice VLAN 30 cutover (8 AudioCodes added; 22 Poly done). AudioCodes required physical power-cycle. Per-phone diagnosis: dropped-calls are RF (band selection). 6 GHz root-caused dark. Optimization master plan built. |
| 2026-06-18 | DESKTOP-TRCIEJA (Lupe Sanchez) perf diagnosed; replace decision. EOL hardware + dual real-time AV. |
| 2026-06-18 | Synology Drive sync architecture diagnosed. Sync-user My Drive only; real shared folders NOT mirrored. Team Folder plan produced. |
| 2026-06-18 | Power outage follow-ups: OpenVPN flapping root-caused (--inactive timeout); kitchen printer straggler resolved. |
| 2026-06-19 | PRODUCTION RF OPTIMIZATION APPLIED (2 AM window) -- 5 GHz retry HALVED. 2.4 power -> MEDIUM on 47 radios. CSCNet BSS-transition ON. Full channel survey (74/74 APs) proved DFS 4-5x cleaner than non-DFS; data-driven clean-DFS plan applied to 72 non-mesh APs (retry 8.7->3.8 avg, 0 co-channel, 0 radar). survey-report.py added. |
| 2026-06-19 | Voice VLAN migration COMPLETE (29/29 Poly) + band-selection diagnosis + Vertical 5 GHz handoff. Howard re-keyed all remaining Poly handsets. 5 GHz-only lock request sent to Richard Turner (Vertical) = last voice item. Kitchen server phone bad (pulled); Bistro relocated to Kitchen; Bistro now has no phone. Billed ticket #32444 (7h), block 55.75->48.75. |
| 2026-06-23 | Planned power outage (05:30-09:00 MST) -- clean shutdown executed + verified. All three core devices self-shut-down on local schedules (independent of tunnel), UPS carried them through. Verified clean at 05:31. Pre-flight confirmed cloud backup SUCCESS + iDRAC/Synology auto-restart backstops. Runbook logged. |
| 2026-06-24 | Syncro ticket review + #32193 Executive share + device-readiness audit + consolidated plan. Built restricted share \\cs-server\Executive (billed 0.5h). Live AD+RMM domain-join diff; 5 on Windows Home blocked. Built docs/REMAINING-WORK-PLAN.md (7 workstreams). Caregivers stay TEST-scoped. |
| 2026-06-24 | CS-SERVER RAID live-verified -- the "degraded/failing" flag was STALE; mirror is healthy. Live Dell OMSA showed both virtual disks Ok, all 5 disks Online/green; the 6/15 degraded self-recovered. 1:0:4 = GLOBAL HOT SPARE. PSU redundancy lost surfaced. No drive pulled; the 2x SSD become a planned upgrade. Service Tag 9MQFTK1. |
| 2026-06-24 | CARF Technology and System Plan deliverable started (Ashley Jensen). First-pass technology-plan packet mapped to the 8 CARF areas; conformance gap analysis + on-brand PDF (via impeccable) + pre-filled intake worksheet. Standing rule: outbound deliverables run through impeccable. |
| 2026-06-24 | CSC ENT device-island consolidation plan (voice + Helpany). Repurpose CSC ENT as a permanent 5 GHz-only WPA2 PPSK island carrying Poly voice (-> VLAN 30) + Helpany radar sensors (-> new VLAN 40); both vendors transition remotely. Unblocks later CSCNet WPA3/6 GHz move. |
| 2026-06-25 | Alma Montt OFFBOARDED (terminated; MC Life Enrichment; no PHI/ALIS). M365: sessions revoked, sign-in blocked, password vaulted, mailbox -> SharedMailbox (Shelby FullAccess+AutoMap), SPB seat freed, hidden from GAL, groups removed. On-prem AD: disabled, moved to OU=Excluded-From-Sync. No litigation hold. Left tenant-security item: Tenant Admin SP holds standing PAA (Graph blocked teardown) -- needs GA removal. |
| 2026-06-25 | Endpoint security migration: Datto EDR/AV rollout + Bitdefender decommission begins. Reconciled 33 GuruRMM devices vs 27 Datto EDR agents; found 8 gaps. Deployed EDR to 6 online clean machines (27->33). Discovered RECEPTIONIST-PC is two physical boxes sharing a hostname (serials MJ0KQH4R, MJ0KQHNP); only one had EDR -- installed on the second (33->34). Removed Bitdefender BEST 8.26.6.644 from both RECEPTIONIST boxes via GravityZone console (API uninstall dead). Cleaned 6 orphaned BD folders. Queued EDR/BD-checks on offline machines; watcher bfm81iqdz left polling. |
| 2026-06-26 | CS-SERVER: full Datto stack removal + SMB "outage" debunked. The endpoint AV was DattoAV (Datto EDR "Endpoint Protection SDK", Bitdefender engine + Avira Sentry), managed by Datto RMM (CentraStage) + Datto EDR Agent (HUNTAgent/Infocyte, tenant azcomp4587) -- NOT GravityZone (so the console removal did nothing). Removed ALL Datto software. The long "SMB error 67" investigation was a TEST-METHOD ARTIFACT -- RMM-dispatched SMB client cmds false-negative even for good targets; CS-SERVER SMB is healthy (Get-SmbSession = 7 users / 30 open files). Karen Rossini share access verified interactively; ALDocs shortcut set. Built the drive-map skill. |
| 2026-06-26 | EDR/Bitdefender straggler 9am pass. BD-checked reconnected machines: DESKTOP-MD6UQI3, Laptop4, SALES4-PC clean; DESKTOP-TRCIEJA = BD_ACTIVE (needs GravityZone console uninstall). NurseAssist queued EDR install confirmed (agent 23c3c36e). DESKTOP-F94M8UT (Alma's, powered off) EDR install queued; DESKTOP-KQSL232 (Lois's old box) decommissioned/removed from list. |
| 2026-06-26 | Windows Home->Pro upgrades continued. NurseAssist + DESKTOP-MD6UQI3 upgraded to Pro for Workstations (changepk edition flip as SYSTEM -> 2-min user reboot warning -> slmgr MAK/ato). Both VOLUME_MAK Licensed. Confirmed NurseAssist is a distinct machine from Assistnurse-pc. SALES4-PC done by supplier. CascadesProxess deferred to a late-night window (access-control appliance in active use). 2 x $99 to bill. |
| 2026-06-29 | ALIS caregiver phone-only + caretaker-only crosscheck. Cross-checked the staff CSV against the live ALIS roster (107 Hired, communityId 622): phone-only caregivers = NONE (only the 3 Transportation drivers are phone-only, and they don't need ALIS); 30 caretaker-only; 2 caregivers carry extra roles (Feller, Nyanzunda); 3 "Care" entries are directors/nurses; 7 caregiver-list people exist in ALIS only as Discharged records (decide reactivate-vs-recreate). |
| 2026-06-30 | Caregiver phone SSO -- license + group + temp-password onboarding. Completed the Entra/identity side for all 40 frontline caregivers: added the 2 missing to SG-Caregivers (now 40), assigned Business Premium (Howard bought 11 more seats -> SPB 34->45 enabled), and set unique forced-change AD temp passwords (vaulted caregiver-temp-passwords-2026-06-30.sops.yaml, delivered via Discord DM). SG-Caregivers corrected to frontline caregivers ONLY (excl. Feller/Nyanzunda). Remaining gate: ALIS Email=UPN match (Howard) + create ALIS records for 7 + AD accounts for 3 ALIS-only caregivers. |
| 2026-06-30 | Tamra Matthews OFFBOARDED (Move-In Coordinator; left June 2026). Cloud-only M365 object: sessions revoked, sign-in blocked, password vaulted, mailbox -> SharedMailbox (Crystal/Megan/Meredith/Ashley FullAccess+AutoMap), O365 Standard seat freed, hidden from GAL, 3 groups stripped. On-prem AD disabled + moved to OU=Excluded-From-Sync. No litigation hold despite PHI-adjacent role (Howard authorized). AutoMapping rollback on rapid grants root-caused (spaced one-at-a-time fix). Follow-up: Megan Hiatt breach re-check. |
| 2026-06-30 | VLAN 20 (CSCNET) printer migration. Migrated Front Desk Epson ET-5800 (.221) + Life Enrichment Canon MF741 (.94) onto VLAN 20 server shares, then dining/chef/medtech/MC-reception. Root-caused a hard blocker: CS-SERVER couldn't reach VLAN 20 printers because the LAN "allow LAN to any" rule policy-routed internal traffic out the WAN (WAN_Group gateway) -- fixed with a top LAN pass rule (gw=default, src CS-SERVER). Established the Point-and-Print policy fix for standard-user driver installs and the Canon UFR-II-only driver requirement (PCL6 -> Error #822). Staged RECEPTIONIST-PC (MemCare box) rename to MEMCARE-STATION. GPO planning doc docs/printer-gpo-map.md created. |
Backlinks
- projects/gururmm -- RECEPTIONIST-PC + CS-SERVER enrolled (site CascadesTucson); fleet EDR/BD + Home->Pro + printer work driven via GuruRMM
- wiki/systems/uos-server -- shared UOS controller hosts the Cascades UniFi site (site_id
685f39068e65331c46ef6dd2); SSH/Mongo access viainfrastructure/uos-server-ssh-key