sync: auto-sync from GURU-5070 at 2026-06-26 06:29:48

Author: Mike Swanson Machine: GURU-5070 Timestamp: 2026-06-26 06:29:48
2026-06-26 06:30:46 -07:00
parent a0b2cfbee1
commit 6d65bff791
4 changed files with 241 additions and 5 deletions
--- a/clients/birth-biologic/session-logs/2026-06/2026-06-26-mike-birthbio-mail-migration-and-datto-vm.md
+++ b/clients/birth-biologic/session-logs/2026-06/2026-06-26-mike-birthbio-mail-migration-and-datto-vm.md
@@ -0,0 +1,193 @@
+## User
+- **User:** Mike Swanson (mike)
+- **Machine:** GURU-5070
+- **Role:** admin
+
+## Session Summary
+
+Worked two BirthBiologic migrations in parallel: the Datto Workplace → SharePoint migration and a
+new Google Workspace → M365 mail migration. The Datto/SharePoint thread started as a "verify current
+state" task but Mike clarified the migration host is a Jupiter VM, not BB-SERVER. Located it as the
+libvirt domain "Windows Server 2016" (actual Windows hostname **ACG-DWP-X-BB**, actually Server 2019
+build 17763) — an ACG-owned migration box running Datto Workplace Server + SPMT, **not enrolled in
+RMM** and sitting on an APIPA address (no LAN). Diagnosed: host bridging was fine (vnet14 enslaved to
+br0, carrier up); the guest simply wasn't getting a DHCP lease from pfSense after ~2 months parked.
+Fixed with a static IP (172.16.3.45/22), installed the GuruRMM agent (enrolled under BirthBiologic /
+Main Office), and confirmed Datto Workplace Server reconnected and is re-syncing. Established (via the
+qemu guest agent and SPMT job storage) that the April 2026 migration only completed Supply Management
+(160 files, custom script) + ITSvcs (excluded); the four large folders (Admin 5.8 GB, Donor Services
+109 GB, Quality 28 GB, Activity Reports) were SPMT's job and last ran 2026-04-29 — completion still
+unconfirmed. Per Mike, full reconciliation waits until Datto finishes re-syncing.
+
+The larger thread was standing up the **Google Workspace → M365 mail migration** end-to-end. Confirmed
+the Google super-admin (`sysadmin@birthbiologic.com`) lives in 1Password (Clients vault item "Google");
+read it via the SOPS-vaulted 1Password service-account token and mirrored it into SOPS. Onboarded
+BirthBio's tenant for **Exchange Operator** (already had Tenant Admin consented, so the suite was
+provisioned programmatically — Exchange Operator SP created + Exchange Administrator role). Pulled the
+authoritative Google roster via domain-wide delegation (20 accounts: 15 active, 5 suspended),
+reconciled against M365, and surfaced two active accounts not on Mike's list (Dr. Chris Gillis
+`medicaldirector@`, Michael Merritt `mmerritt@`) plus an address mismatch (Mindi is `mindim@` in
+Google, `mmaher@` in M365).
+
+Provisioned the M365 target side to Mike's licensing rules: active-12 → Business Premium (assigned BP
+to Mei Mei + Valerie, freed Savanna's BP by moving her to Exchange-only); created Gillis + Merritt with
+Exchange-only and vaulted their passwords; licensed the 4 disabled former employees with Exchange-only
+(kept sign-in disabled) as future shared-mailbox targets. License math closed exactly: 14 Business
+Premium + 7 Exchange Online Plan 1, all consumed.
+
+Hit a real blocker creating the Gmail migration endpoint: Google returned `unauthorized_client … not
+authorized for any of the scopes requested`. Root cause = the DWD grant had only 3 of Microsoft's
+required **5** scopes (missing `m8/feeds` and `gmail.settings.sharing`); Google rejects the migration
+token all-or-nothing. Verified the exact 5-scope string against live MS Learn + a Grok live-search
+cross-check (Gemini CLI was down on this box), updated our runbook, and Mike re-authorized all 5 in the
+BB Google console. After that the endpoint (`BB-Gmail`) created cleanly and **Batch 1 (14 live
+mailboxes, mail + calendar + contacts) was created and auto-started — Status: Syncing**.
+
+## Key Decisions
+
+- **Datto VM gets a static IP (172.16.3.45), not a pfSense DHCP fix.** The fault was pfSense not
+  leasing this MAC after a long park; a static on the ACG server range (172.16.3.x) is the reliable,
+  convention-consistent fix. Follow-up: add a pfSense reservation or confirm it's outside the DHCP pool.
+- **Enrolled the Datto VM under BirthBiologic / Main Office** (not AZ Computer Guru) since the box exists
+  solely for BirthBio's migration and we had that site key; reversible (agents can be moved).
+- **Former employees migrate to shared mailboxes via a temp Exchange-only license** (migrate into a
+  licensed mailbox → convert to shared ≤50 GB = free → reclaim license). Source Google accounts must be
+  **un-suspended** during migration (Gmail API can't read suspended accounts).
+- **Licensing tiers (Mike's rules):** active-users-list → Business Premium; live Google accounts not yet
+  in M365 (Gillis, Merritt) → Exchange-only ("E1" = Exchange Online Plan 1); formers → Exchange-only
+  (reclaimable). `operations@` stays BP through migration. Existing BP users left on BP (not downgraded).
+- **Batch sequencing:** live users first (Batch 1); formers as Batch 2 after un-suspending them in Google
+  and freeing Workspace seats by suspending already-migrated live users.
+- **Mindi mapped via the CSV `Username` column** (`EmailAddress=mmaher@`, `Username=mindim@`) — the
+  proper MS mechanism — plus a belt-and-suspenders `mindim@` proxy on her mailbox.
+- **Target delivery domain = `birthbiologic.onmicrosoft.com`** for Batch 1 (no routing subdomain exists;
+  acceptable for a near-term cutover; MS prefers a subdomain for long coexistence).
+- **Drove Exchange via REST `InvokeCommand`** (Exchange Operator app token) — the EXO PowerShell module
+  isn't installed and the app has no vaulted cert, so app-only Connect-ExchangeOnline wasn't available.
+
+## Problems Encountered
+
+- **Datto VM on APIPA (no LAN).** Host bridging fine; pfSense wasn't leasing the MAC. Fixed with static
+  172.16.3.45/22, GW 172.16.0.1, DNS 172.16.0.1+1.1.1.1. Verified gateway/internet/DNS + RMM check-in.
+- **`vault.sh get-field` returned `null` (len 4)** for nested secrets until the field arg used dotted
+  path: `credentials.client_secret`, `credentials.credential`. Plain leaf names don't resolve.
+- **SPB skuId mismatch.** The scope doc's BP GUID (`cbdc14ab-d96c-4132-b7f4-1f3a3a819bb4`) was stale; the
+  tenant's real SPB skuId is `cbdc14ab-d96c-4c30-b9f4-6ada7cdc1d46`. License assign 400'd until corrected.
+- **License seat propagation lag** — Valerie's BP assign 400'd ("no available licenses") immediately after
+  freeing Savanna's seat; succeeded on retry seconds later.
+- **`proxyAddresses` read-only via Graph** — adding Mindi's alias required Exchange `Set-Mailbox` (EXO),
+  not a Graph PATCH.
+- **Gmail migration endpoint failed: `unauthorized_client … not authorized for any of the scopes
+  requested`.** DWD had 3 of 5 required scopes. Got the verbatim 5-scope string from MS Learn + Grok;
+  Mike re-authorized; endpoint then created.
+- **`onboard365.sh` vault path** — looked at `/c/Users/guru/.claude/identity.json`; fixed by exporting
+  `VAULT_ROOT_ENV=/d/vault` (logged as friction).
+- **GCP API enable initially run as the wrong identity** — Mike first ran `gcloud services enable` as
+  `sysadmin@birthbiologic.com` (no rights to ACG's project); succeeded once run as the ACG owner of
+  `acg-msp-access`.
+- **Gemini CLI down** (`throwIneligibleOrProjectIdError`, needs interactive re-login) — used Grok for the
+  live-doc cross-check instead. Logged to errorlog.
+
+## Configuration Changes
+
+- **ACG-DWP-X-BB (Jupiter "Windows Server 2016" VM):** static IP 172.16.3.45/22, GW 172.16.0.1, DNS
+  172.16.0.1 + 1.1.1.1 (persistent). GuruRMM agent installed (universal installer), enrolled BirthBiologic
+  / Main Office, agent `a4524e85-8a07-45d0-91b1-51ce7e2ca74a`.
+- **BirthBio M365 tenant (19a568e8-…):** onboarded Exchange Operator (+ Defender Add-on) SPs via
+  `onboard365.sh provision`; roles assigned (Exchange Admin on Exchange Operator + Security Investigator,
+  CA Admin on Tenant Admin, User Admin + Auth Admin on User Manager).
+  - License changes: Mei Mei (`msenthavy`) +BP; Valerie (`vvaneaton`) +BP; Savanna (`sabron`) BP→EXO;
+    created `medicaldirector@` (Gillis) +EXO and `mmerritt@` (Merritt) +EXO; licensed `aboutte`, `araso`,
+    `khoffman`, `pnelson` with EXO (kept sign-in disabled).
+  - `Set-Mailbox mmaher@` added secondary `smtp:mindim@birthbiologic.com`.
+  - Created Gmail migration endpoint `BB-Gmail`; created + auto-started migration batch `BB-Batch1` (14
+    users, TargetDeliveryDomain `birthbiologic.onmicrosoft.com`, NotificationEmails sysadmin@).
+- **Vault (pushed):** `clients/birth-biologic/google-workspace.sops.yaml`,
+  `clients/birth-biologic/m365-medicaldirector.sops.yaml`, `clients/birth-biologic/m365-mmerritt.sops.yaml`.
+- **Repo:** updated `projects/msp-tools/runbooks/google-workspace-to-m365-migration.md` (exact 5-scope
+  string + all-or-nothing gotcha + Contacts-API-retired/People-API + GCP-owner notes).
+- **errorlog.md:** gemini CLI failure entry (+ onboard365 vault-path friction).
+
+## Credentials & Secrets
+
+- **Google Workspace super-admin** `sysadmin@birthbiologic.com` (source tenant) — sourced from 1Password
+  Clients vault item "Google"; mirrored to SOPS `clients/birth-biologic/google-workspace.sops.yaml`
+  (`credentials.password`, 19 chars). Used for admin.google.com console (DWD/API) + as the migration
+  impersonation admin.
+- **M365 mailbox — Dr. Chris Gillis** `medicaldirector@birthbiologic.com` — created this session; password
+  vaulted at `clients/birth-biologic/m365-medicaldirector.sops.yaml` (forceChangePasswordNextSignIn=true).
+- **M365 mailbox — Michael Merritt** `mmerritt@birthbiologic.com` — created this session; password vaulted
+  at `clients/birth-biologic/m365-mmerritt.sops.yaml` (forceChangePasswordNextSignIn=true).
+- App secrets used (already vaulted): Tenant Admin `msp-tools/computerguru-tenant-admin`
+  (`credentials.client_secret`); Exchange Operator `msp-tools/computerguru-exchange-operator`
+  (`credentials.client_secret`); Google SA `msp-tools/acg-msp-access-google-workspace`
+  (`credentials.credential`, full JSON); 1Password service token
+  `infrastructure/1password-service-account.sops.yaml`.
+
+## Infrastructure & Servers
+
+- **ACG-DWP-X-BB** — Jupiter libvirt domain "Windows Server 2016" (actually WS2019, build 17763). Windows
+  hostname ACG-DWP-X-BB. NIC virtio 52:54:00:d4:8e:59 on br0 (vnet14). Static 172.16.3.45/22. Runs Datto
+  Workplace Server (svc `datto_workplace_server.default`, proc WorkplaceServer) + SPMT (under
+  Administrator profile). RMM agent `a4524e85-8a07-45d0-91b1-51ce7e2ca74a`. Datto source tree
+  `C:\Users\Public\Desktop\Datto Workplace Server Projects`.
+- **Jupiter** 172.16.3.20 (Unraid, virsh host). LAN 172.16.0.0/22, GW pfSense 172.16.0.1. guest-exec
+  helper at `/root/gx.sh` on Jupiter.
+- **BB-SERVER** — RMM agent `6c02baa7-0f1c-4990-b466-c9ab9eaefd3b`. Also has Datto Workplace Server + the
+  original custom-script artifacts at `C:\GuruMigration` (bb-migration-state.json shows 160 Supply Mgmt +
+  49 ITSvcs uploaded in April).
+- **BirthBio M365 tenant** `birthbiologic.com` / `19a568e8-9e88-413b-9341-cbc224b39145`.
+  - SPs: Tenant Admin `7a199b11-97fb-4e65-917d-f8d29a53ba49`; Exchange Operator
+    `bab4699b-32a3-4434-9cad-7a4a08cc4d9e`; Security Investigator `bf684a4b-…`; User Manager `3347ebcc-…`;
+    Defender Add-on `161b8f61-…`. New user objects: Gillis `1bd491e1-3ba6-4214-8c6d-46426f8681da`, Merritt
+    `117a3367-cd5f-4565-af11-af5ff089224f`.
+  - SKUs: Business Premium (SPB) `cbdc14ab-d96c-4c30-b9f4-6ada7cdc1d46` (14/14); Exchange Online Plan 1
+    (EXCHANGESTANDARD) `4b9405b0-7788-4568-add1-99614e613b69` (7/7). Accepted domains: birthbiologic.com
+    (default), birthbiologic.onmicrosoft.com.
+- **Google project** `acg-msp-access` (number 806899474449). SA `acg-msp-access@acg-msp-access.iam.gserviceaccount.com`,
+  OAuth2 client ID `102231607889615995452`. APIs enabled: Gmail, Calendar (calendar-json), People.
+- **Google roster (DWD pull):** 15 active, 5 suspended. Active staff emails per `clients/birth-biologic/
+  docs/migration/google-to-m365-scope.md`; Mindi = `mindim@` (Google) ↔ `mmaher@` (M365).
+
+## Commands & Outputs
+
+- **Required Google DWD scopes (exact, 5, comma-separated, no spaces):**
+  `https://mail.google.com/,https://www.googleapis.com/auth/calendar,https://www.google.com/m8/feeds/,https://www.googleapis.com/auth/gmail.settings.sharing,https://www.googleapis.com/auth/contacts`
+  (`m8/feeds` is a still-valid alias for the contacts scope, served by People API; legacy Contacts API
+  retired 2022, not enableable, not needed.)
+- EXO via REST: `POST https://outlook.office365.com/adminapi/beta/{tenant}/InvokeCommand` with Exchange
+  Operator app token (`scope=https://outlook.office365.com/.default`), body
+  `{"CmdletInput":{"CmdletName":"…","Parameters":{…}}}`. byte[] params (ServiceAccountKeyFileData, CSVData)
+  passed as **base64 strings**.
+- `New-MigrationEndpoint -Gmail -Name BB-Gmail -ServiceAccountKeyFileData <b64> -EmailAddress sysadmin@birthbiologic.com` → created.
+- `New-MigrationBatch -Name BB-Batch1 -SourceEndpoint BB-Gmail -CSVData <b64> -TargetDeliveryDomain birthbiologic.onmicrosoft.com -AutoStart -NotificationEmails sysadmin@` → Status=Syncing, Total=14.
+- Get-MigrationUser BB-Batch1 → 14 Provisioning, 0 skipped (normal initial state).
+- Datto source counts (ACG-DWP-X-BB): Admin 6,279/5.8GB · Donor Services 56,826/109GB · Quality 3,714/28GB
+  · Supply Mgmt 160/33MB · Activity Reports 1 · ITSvcs 52 (excluded).
+
+## Pending / Incomplete Tasks
+
+- **Batch 1 monitor → MX cutover.** Watch `BB-Batch1` Provisioning→Syncing→Synced. When Synced: flip MX in
+  SiteGround → M365, update SPF (`include:spf.protection.outlook.com`), enable/publish DKIM (2 CNAMEs),
+  autodiscover CNAME → autodiscover.outlook.com, run final delta, then **complete** the batch.
+- **Batch 2 — 5 former employees → shared.** Un-suspend each in Google (free Workspace seats by suspending
+  migrated live users), run a Gmail batch (targets already EXO-licensed: aboutte, araso, khoffman, pnelson,
+  sabron), then convert to shared mailboxes and reclaim the 5 EXO licenses.
+- **Datto → SharePoint reconciliation.** After ACG-DWP-X-BB finishes re-syncing with Datto cloud, compare
+  source vs each SharePoint site to confirm what the April SPMT run left unfinished (Admin / Donor Services
+  / Quality / Activity Reports).
+- **pfSense:** add a DHCP reservation for 172.16.3.45 (MAC 52:54:00:d4:8e:59) or confirm it's outside the pool.
+- **Valerie VanEaton** — active (receiving daily; last *sent* 2026-05-13). Julie to confirm whether the
+  mid-May send drop-off = leave/departure; if departed, move her to the former→shared track.
+- **Decisions still open:** confirm Merritt's long-term tier; whether `operations@` becomes shared post-migration.
+- **Wiki:** BirthBio article is stale (says migration incomplete / 13 mailboxes) — recompile.
+
+## Reference Information
+
+- Migration scope doc: `clients/birth-biologic/docs/migration/google-to-m365-scope.md`.
+- Runbook (updated): `projects/msp-tools/runbooks/google-workspace-to-m365-migration.md`.
+- MS Learn: `manually-configuring-gsuite-for-migration` (scope string), `automated-migration-neweac`,
+  `google-workspace-migration-prerequisites`, `perform-g-suite-migration`.
+- RMM install one-liner (BirthBio site): `irm https://rmm.azcomputerguru.com/install/BRIGHT-PEAK-5980/windows | iex`.
+- Discord DMs to Mike: message_id 1520034139900739627 (initial DWD), 1520055625302675537 (corrected 5-scope).
+- Vault enrollment key: `clients/birth-biologic/gururmm-site-main` (site BRIGHT-PEAK-5980, id 3b20ef97-…).
--- a/errorlog.md
+++ b/errorlog.md
@@ -17,6 +17,18 @@ Categories (the `[type]` tag): _(none)_ = skill/command execution failure ·

 <!-- Append entries below this line -->

+2026-06-26 | GURU-5070 | agy/gemini | gemini CLI headless failed: throwIneligibleOrProjectIdError / _doSetupUser (auth-eligibility, needs interactive re-login) [ctx: task=verify-gws-migration-scopes]
+
+2026-06-26 | GURU-5070 | agy | gemini returned no response (empty after 3 attempts) [ctx: mode=search err=    at process.processTicksAndRejections (node:internal/process/task_queues:104:]
+
+2026-06-26 | GURU-5070 | remediation-tool | onboard-tenant: grant_app_role appRoleAssignment failed [ctx: role=bf394140-e372-4bf9-a898-299cfc7564e5 msg=Resource '161b8f61-5c16-4e1a-9a23-4bb7076b0946' does not exist or one of its que]
+
+2026-06-26 | GURU-5070 | remediation-tool | onboard-tenant: grant_app_role appRoleAssignment failed [ctx: role=6931bccd-447a-43d1-b442-00a195474933 msg=Resource 'bab4699b-32a3-4434-9cad-7a4a08cc4d9e' does not exist or one of its que]
+
+2026-06-26 | GURU-5070 | remediation-tool | onboard-tenant: grant_app_role appRoleAssignment failed [ctx: role=df021288-bdef-4463-88db-98f22de89214 msg=Resource 'bab4699b-32a3-4434-9cad-7a4a08cc4d9e' does not exist or one of its que]
+
+2026-06-26 | GURU-5070 | remediation-tool | onboard-tenant: failed to acquire Tenant Admin token [ctx: tenant=19a568e8-9e88-413b-9341-cbc224b39145 exit=3]
+
 2026-06-26 | Howard-Home | drive-map | drive-map verify failed on DESKTOP-LPOPV30 [ctx: cmd=e932bc94-0557-4913-a0b1-c97c1aa5da26]

 2026-06-26 | Howard-Home | drive-map | drive-map verify failed on DESKTOP-LPOPV30 [ctx: cmd=18fec38b-8fae-4a1b-a3d8-5b90b124dbc2]
--- a/projects/msp-tools/runbooks/google-workspace-to-m365-migration.md
+++ b/projects/msp-tools/runbooks/google-workspace-to-m365-migration.md
@@ -48,8 +48,14 @@ ACG already has a Google service account for Workspace access:
 ## 4. MS native migration — end to end

 **Step 1 — Source (Google) prep**
-1. In **GCP** (project `acg-msp-access` or a new one): ensure the service account exists and a JSON key is in the vault. Enable APIs: **Gmail, Google Calendar, Google People (Contacts), Admin SDK (Directory)**.
-2. In the SOURCE **Google Admin console** → Security → API controls → **Domain-wide delegation** → add the SA **Client ID** with the Microsoft-required OAuth scopes (Gmail/Calendar/Contacts/Directory — copy the exact scope list from the EAC migration wizard so they match).
+1. In **GCP** (project `acg-msp-access` or a new one): ensure the service account exists and a JSON key is in the vault. Enable APIs: **Gmail API, Google Calendar API, People API**. (The legacy *Contacts API* was retired by Google in 2022 and **cannot be enabled** — the `m8/feeds` contacts scope is now an alias served by the People API, so People API enablement covers it. Enabling the APIs in `acg-msp-access` requires being signed in as the **ACG owner** of that project — a *client* super-admin has no rights to ACG's GCP project.)
+2. In the SOURCE **Google Admin console** → Security → API controls → **Domain-wide delegation** → add the SA's **OAuth2 Client ID** (the SA's numeric "Unique ID", NOT the app client_id) with the **exact 5-scope string below, comma-separated, no spaces**. Google rejects the migration token request **all-or-nothing** — if even one scope is missing the endpoint fails later with `unauthorized_client … not authorized for any of the scopes requested`. Verified current 2026-06 (MS Learn `manually-configuring-gsuite-for-migration` + Grok live cross-check):
+
+   ```
+   https://mail.google.com/,https://www.googleapis.com/auth/calendar,https://www.google.com/m8/feeds/,https://www.googleapis.com/auth/gmail.settings.sharing,https://www.googleapis.com/auth/contacts
+   ```
+
+   Propagation can take 15 min–24 h (usually minutes). Do NOT rely on a smaller "mail+calendar+contacts" set — `m8/feeds` and `gmail.settings.sharing` are both required by the MS endpoint.
 3. Confirm a Google super-admin mailbox exists for the migration to impersonate.

 **Step 2 — Target (M365) prep**
--- a/wiki/systems/uos-server.md
+++ b/wiki/systems/uos-server.md
@@ -2,10 +2,11 @@
 type: system
 name: uos-server
 display_name: UOS Server (UniFi OS Server)
-last_compiled: 2026-06-21
-compiled_by: HOWARD-HOME/claude-main
+last_compiled: 2026-06-26
+compiled_by: GURU-5070/claude-main
 sources:
  - session-logs/2026-06/2026-06-21-howard-unifi-pfsense-control-verbs.md
+  - 2026-06-26 Rocky 9.1->9.8 host OS patch (mike)
 backlinks:
  - systems/jupiter
  - systems/pfsense
@@ -23,7 +24,7 @@ backlinks:

 ## What / where it is

- **Guest:** Rocky Linux 9.1, hostname-internal "UOS Server". Guest IP **`172.16.3.29`** (ACG internal LAN).
+- **Guest:** Rocky Linux **9.8** (kernel `5.14.0-687.17.1.el9_8`; last patched 2026-06-26), hostname-internal "UOS Server". Guest IP **`172.16.3.29`** (ACG internal LAN).
 - **Hypervisor:** Jupiter (`172.16.3.20`, Unraid) — virsh domain **`Unifi`** (id 1). `virsh list` to confirm running.
 - **App stack (inside guest):** UniFi Network = `ace.jar` (Java) + classic **MongoDB `ace`** on `127.0.0.1:27117`, plus `unifi-core` (Postgres) for UniFi-OS identity/integration. All of it runs **inside a rootless podman container `uosserver`** (host user `uosserver`, uid 1000) — so the app files and mongo are NOT on the guest rootfs.
 - **ui.com cloud:** host id `2d6b654d-9b79-4eaa-b2e1-52062a5690ef` in the Site Manager account.
@@ -85,6 +86,30 @@ There is **no mongo client on the guest host**; the shell is `/usr/bin/mongo` *i
 - **`rogue`** — neighbor/over-the-air BSSIDs seen by APs. **Not ACG gear** — a MAC hit here is someone else's WiFi, ignore it for device hunts.
 - **Pending/unadopted devices:** the controller only persists a discovered device into `device` with `adopted:false`. If `db.device.count({adopted:false})` is `0`, there are **no** pending devices controller-wide — an "unadopted" device that returns nothing here simply has not reached this controller (not on a network it can discover, or managed by a different console). The cloud API and integration API show adopted gear only, so they cannot find it either; locating it then needs L2/DHCP/ARP on the gateway of the site it is physically cabled to.

+## Host OS maintenance (Rocky)
+
+The UniFi app self-updates (the `uosserver-updater.service` rebuilds the rootless podman
+container; UOS Server **5.1.19** / Network **10.4.57** as of 2026-06-26). The **host OS is
+NOT auto-patched** — it must be updated manually over SSH. The two layers are independent:
+patching the host does not touch the container/UniFi data (named podman volumes persist).
+
+**Procedure** (root via the fleet SSH key):
+```bash
+ssh -i <key> root@172.16.3.29 'dnf -y update'   # ~few min; kernel updates need a reboot
+ssh -i <key> root@172.16.3.29 'systemctl reboot' # controller drops briefly; APs keep serving WiFi
+# verify after ~3-4 min:
+ssh -i <key> root@172.16.3.29 'uosserver status'                  # container Up (healthy)
+ssh -i <key> root@172.16.3.29 'curl -sk -o /dev/null -w "%{http_code}\n" https://127.0.0.1:11443/'  # 200
+```
+- **Safety net:** daily UniFi auto-backups live at
+  `~uosserver/.local/share/containers/storage/volumes/uosserver_var_lib_unifi/_data/backup/autobackup/`
+  (newest `autobackup_<ver>_<date>.unf`) — survives the OS update. For the OS layer itself,
+  a virsh snapshot of the **"Unifi"** VM on [[jupiter]] is the rollback point (optional).
+- **Reboot impact:** only central management/stats drop while the VM reboots (~30-60s back to
+  SSH, ~3-4 min to container healthy); APs/switches keep forwarding traffic the whole time.
+- **History:** 2026-06-26 — Rocky **9.1 -> 9.8** (362 pkgs, kernel `162.6.1.el9_1` ->
+  `687.17.1.el9_8`, full security backlog cleared); clean reboot, controller back healthy.
+
 ## Related tooling — pfSense gateway layer (works together)

 This UOS controller and the **pfSense gateway tooling** are the two halves of the **`unifi-wifi`