# Session Log — Pavon — 2026-04-29

## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-BEAST-ROG
- **Role:** admin

---

## Mode
`infra` — OwnCloud VM diagnostics on Jupiter, multi-host SSH, scheduled cron edits.

---

## Session Summary

A performance issue was reported on a Windows Server 2016 VM running SharePoint Migration Tool for BirthBiologic, hosted on Jupiter (Unraid 172.16.3.20). Initial analysis incorrectly attributed high CPU usage to CPU pinning, but further investigation revealed that the high load was caused by the OwnCloud VM (172.16.3.22) instead. The OwnCloud VM, running Rocky Linux 9.6, was experiencing a severe performance degradation due to an excessive number of stalled `occ system:cron` processes. These processes were executing inefficient MariaDB queries that were scanning the `oc_filecache` table repeatedly, leading to sustained high CPU usage and load average. The inefficient query pattern used a wildcard in the LIKE clause, incorrect collation, and missing indexes, resulting in full table scans.

After killing the stalled processes and implementing a flock lock to prevent future stacking, CPU usage dropped significantly. Further investigation into the storage configuration and data retention policy confirmed that OwnCloud is the source of truth for the data (3-year retention), and no immediate migration or cleanup was performed at the user's request. Credentials access was switched from 1Password to SOPS partway through the session per user preference, which uncovered drift between the two sources for the OwnCloud VM root password.

---

## Key Decisions

- **Kill stalled cron processes immediately** rather than waiting them out — system was in deep distress (load avg 80, ~70 tasks queued waiting on CPU) and there was no risk to user data
- **Wrap cron with `flock -n`** instead of disabling cron outright — preserves OwnCloud's normal background-job processing while making the stacking-spiral structurally impossible
- **Defer per-user versioning disable** — OwnCloud Community refused `--groups` flag for `files_versions`. Workaround identified (`occ versions:cleanup pavon` on a daily cron) but deferred per user request
- **Did not bypass OwnCloud for ingest** — user clarified GeoVision NVRs at Curves and Raiders sites use OwnCloud Desktop sync client with virtual file placeholders to save NVR-local disk; NVR units lack direct SMB access to Jupiter/Saturn, so the existing ingest path stays
- **Defer the migration/retention scripts** — current state (cron-flock + status quo on versioning) is stable enough; user wants to revisit the architectural cleanup with more time

---

## Problems Encountered

- **Incorrect initial diagnosis of CPU pinning** — first hypothesis was that the Windows VM's 16 vCPUs were oversubscribed onto 8 host threads. `virsh dumpxml` showed no `<cputune>` block at all on the Windows VM. Re-pivoted by checking which qemu process actually owned the load, found OwnCloud
- **Stale `occ system:cron` processes** — 75-126 piled up since 2026-04-27. No flock wrapper meant every 15-min tick fired a new run while previous ones still ground on. Resolved with `pkill -f 'occ system:cron'` then `pkill -9` for stragglers
- **Inefficient MariaDB query pattern** in OwnCloud's filecache scanner — `name COLLATE utf8mb4_general_ci LIKE 'pattern%middle%suffix'` defeats the `utf8mb4_bin` index and the wildcard mid-string. The schema only has `(parent, name)` and `(storage, path_hash)` indexes, no `(storage, name)`. Each query becomes a 257K-row scan on storage 78
- **Failed attempt to disable versioning for one user** — `app:enable files_versions --groups <group>` rejected with "files_versions can't be enabled for groups." Reverted by re-enabling globally
- **Credential drift between 1Password and SOPS** — SOPS had `r3tr0gradE99!!` for OwnCloud VM root, 1Password had `Paper123!@#-unifi!`. SOPS value worked, 1Password did not. Worth reconciling
- **PowerShell mangling shell metachars** — repeatedly tripped on `$()`, `*`, `(`, backticks when invoking SSH inline. Worked around by writing scripts to local files, scp-ing, then executing remotely

---

## Credentials Used / Discovered

### Jupiter (Unraid Primary)
- **Host:** 172.16.3.20
- **User:** root
- **Password (1Password and SOPS agree):** `Th1nk3r^99##`
- **SOPS path:** `infrastructure/jupiter-unraid-primary.sops.yaml`
- **1Password item id:** `5ji4rsgvn6feare6fahxsqauui`

### OwnCloud VM (Rocky Linux 9.6)
- **Host:** 172.16.3.22
- **Hostname:** cloud.acghosting.com
- **User:** root
- **Password (SOPS — works):** `r3tr0gradE99!!`
- **Password (1Password — STALE, does NOT work):** `Paper123!@#-unifi!`  ← **needs reconciliation**
- **SOPS path:** `infrastructure/owncloud-vm.sops.yaml`
- **1Password item id:** `h6usgzxxn26kvckxz5dhssxdai`
- **SSH host key (ed25519):** `SHA256:Yy4oFv5HudmKjNJ4IZgHcuSSmeBvUg+ZJta6iLasdqU`

### MariaDB on OwnCloud VM
- Local socket auth as root from the VM (no password prompt for `mysql` CLI when run as root)
- OwnCloud's app DB: `owncloud` schema, user `owncloud@localhost`

---

## Infrastructure Touched

| Host | IP | Role | Action |
|---|---|---|---|
| Jupiter (Unraid Primary) | 172.16.3.20 | Hypervisor | Read-only diagnostics: `virsh list`, `virsh dumpxml`, `virsh vcpuinfo`, `mpstat`, `ps` |
| OwnCloud VM | 172.16.3.22 | Rocky Linux 9.6, runs OwnCloud + MariaDB 10.5.29 | `pkill` cron processes, edited apache crontab, group/app changes (reverted) |
| Uranus (Unraid Secondary) | 172.16.3.21 | SMB share host (`Storage` share) | None this session — referenced as future archive target |

### VM cputune findings on Jupiter

- **Windows Server 2016 VM:** No CPU pinning. Topology `sockets=1, cores=8, threads=2` (16 vCPUs). Memory 16 GB current / 32 GB max with ballooning.
- **OwnCloud VM:** vCPUs 0-7 pinned 1:1 to host CPU 0,28,1,29,2,30,3,31 — exactly the eight LPs that were saturated. Topology `cores=4, threads=2` (8 vCPUs).

### OwnCloud users and groups

- 10 users total: Martell, anaise, bst, jburger, mara, minrec, **pavon**, rohrbach, sysadmin, themarcgroup
- Existing groups: ACG, Clients, PST, QMS, Stamback, Stoltz, WCP, admin
- Only sysadmin is in any group (ACG, admin); all other users are unaffiliated

### Storage layout (OwnCloud VM)

- **OwnCloud data dir:** `/owncloud`, NFS-mounted from `172.16.3.20:/mnt/user/OwnCloud`
- **Filesystem state:** 932 GB total, 677 GB used, 248 GB free, **74% full**
- **Pavon's storage (numeric_id 78, `home::pavon`):**
  - `/owncloud/pavon/files/Curves/` — 188,920 files (all sub-paths Curves/Data-F/CamNN/YYYYMMDD/Event*.Avi or .Wav)
  - `/owncloud/pavon/files/Raiders/` — 48,978 files (Raiders/Cameras{,2}/CamNN/YYYYMMDD/Event*.Avi)
  - Two NVR log files at root (`NVR-18019140.out`, `NVR-18082322.out`, ~16K each)
  - **Total ~237K files**
  - **30 GB of accumulated junk versions** in `/owncloud/pavon/files_versions/` (1,326 version files, 1,383 filecache rows)
- **External storage (numeric_id 6, mount_id 6):**
  - Mount point in pavon's view: `/Archive`
  - Backend: SMB Personal (unique file IDs)
  - Host: 172.16.3.21 (Uranus)
  - Share: `Storage`
  - SMB user: `owncloud`
  - **`filesystem_check_changes` already set to 0** — OwnCloud doesn't auto-rescan this on cron

### File age distribution (pavon)

```
2024:      1 file (oldest from 2024-12-21)
2025: 162,898 files
2026:  74,719 files
Older than 365 days: 256 files (as of 2026-04-29)
```

---

## Architecture (NEW context from this session)

- **GeoVision NVR** units at Curves and Raiders client sites
- Each NVR runs **OwnCloud Desktop sync client** with the sync folder pointed at the NVR's data drop directory
- After upload, OwnCloud client converts the local file to a **virtual file placeholder** to conserve NVR-local disk; if the NVR ever needs the file, the placeholder triggers a re-download
- **NVR units have NO direct SMB access** to Jupiter or Saturn — they reach OwnCloud only via the WebDAV interface used by the desktop client
- **OwnCloud is the source of truth** for this footage, not a backup of it
- **Retention policy: 3 years**. Older may be deleted
- **Pavon never uses OwnCloud directly** — only the NVR interface for footage retrieval
- **GeoVision has no built-in age-based file routing** — can't move old files to a different folder on the NVR side

---

## Commands & Outputs (Critical)

### Identifying the runaway VM (on Jupiter)

```bash
ssh root@172.16.3.20 'ps -eo pid,pcpu,pmem,comm,args --sort=-pcpu | head -3'
# PID 15343  486% CPU  qemu-system-x86_64 ... guest=OwnCloud
# PID 2349755 118% CPU  qemu-system-x86_64 ... guest=Windows Server 2016
# PID 13887   25% CPU  qemu-system-x86_64 ... guest=Unifi

mpstat -P 0,1,2,3,28,29,30,31 1 1
# All eight LPs showed %guest near 100, %idle near 0 — load was guest VM, not host
```

### OwnCloud cputune confirming pin (on Jupiter)

```bash
virsh dumpxml OwnCloud | grep -A 10 cputune
# <vcpupin vcpu='0' cpuset='0'/>
# <vcpupin vcpu='1' cpuset='28'/>
# ...
# <vcpupin vcpu='7' cpuset='31'/>
```

### Stale cron count and load (on OwnCloud VM)

```bash
ps -ef | grep 'occ system:cron' | grep -v grep | wc -l
# 126
uptime
# load average: 80.38, 77.60, 76.98
```

### The bad query pattern (sample)

```sql
SELECT `fileid`, `storage`, `path`, `parent`, `name`, `mimetype`, `mimepart`,
       `size`, `mtime`, `encrypted`, `etag`, `permissions`, `checksum`
FROM `oc_filecache`
WHERE `storage` = '78'
  AND `name` COLLATE utf8mb4_general_ci LIKE 'Event20260412190705025.Wav.v%.d1776373045'
```

### oc_filecache schema confirming the index situation

```
CREATE TABLE oc_filecache (
  ...
  `name` varchar(250) DEFAULT NULL,
  PRIMARY KEY (`fileid`),
  UNIQUE KEY `fs_storage_path_hash`   (`storage`, `path_hash`),
  KEY `fs_parent_name_hash`           (`parent`,  `name`),
  KEY `fs_storage_mimetype`           (`storage`, `mimetype`),
  KEY `fs_storage_mimepart`           (`storage`, `mimepart`),
  KEY `fs_storage_size`               (`storage`, `size`, `fileid`),
  KEY `fs_parent_storage_size`        (`parent`,  `storage`, `size`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
```

No `(storage, name)` index exists. Combined with the `COLLATE utf8mb4_general_ci` override and mid-string wildcard, every query becomes a full scan of the 257K rows in storage 78.

### Killing the cron stack

```bash
pkill -f 'occ system:cron'
sleep 3
pkill -9 -f 'occ system:cron'
# Connection dropped briefly during the kill due to system shock — expected
# Reconnected: 0 stale processes; load avg dropped 80 -> 27 -> 5 within ~3 minutes
```

### Active queries after cleanup

```sql
SELECT COUNT(*) FROM information_schema.PROCESSLIST WHERE COMMAND != 'Sleep';
-- 1  (just my own query)
```

---

## Configuration Changes (PERSISTED)

### OwnCloud VM — apache crontab (`/var/spool/cron/apache`)

**Before (caused stacking):**
```cron
*/15  *  *  *  * /usr/bin/php -f /var/www/owncloud/occ system:cron
```

**After (current production):**
```cron
*/15 * * * * /usr/bin/flock -n /tmp/oc-cron.lock /usr/bin/php -f /var/www/owncloud/occ system:cron
```

`flock -n` makes new ticks bail immediately if the previous run still holds the lock — at most one `occ system:cron` ever runs.

**Backup:** `/root/apache-crontab.backup-20260428-pre-flock` (67 bytes, contains the original line)

### Group changes (NOT REVERTED)

A group named **`versioning_users`** was created on the OwnCloud VM during the failed per-user-versioning attempt, with all 9 non-pavon users added. The intent was to scope `files_versions` to that group, but `app:enable --groups` was rejected. The group **still exists** with those memberships. It's harmless (no app uses it) but worth knowing about for next session — could be deleted with:

```bash
sudo -u apache php /var/www/owncloud/occ group:delete versioning_users
```

### `files_versions` app

Was DISABLED briefly during the failed attempt; re-ENABLED globally to restore status quo. Currently enabled for all users as before.

---

## Pending / Incomplete Tasks

All deferred per user request. The system is stable now thanks to the flock fix; the items below are improvements, not emergencies.

| # | Task | Status | Notes |
|---|---|---|---|
| 1 | Investigate pavon storage layout | DONE | Findings captured above |
| 2 | Disable versioning for pavon | DEFERRED | Approach: daily cron `occ versions:cleanup pavon` + `occ trashbin:cleanup pavon` (also clean existing 30 GB of pavon's accumulated versions) |
| 3 | Set up external storage mount for archive | DONE (already existed) | Storage 6, /Archive, SMB to Uranus, filesystem_check_changes already 0 |
| 4 | Disable trash on external archive storage | DEFERRED | Mooted by retention design — `find -mtime +1095 -delete` bypasses OwnCloud trash anyway |
| 5 | Build monthly migration cron (internal → /Archive) | DEFERRED | NVR architecture forces files to land internal first. Cutoff target: 90 days. Caveat: `/Archive` uses "SMB Personal (unique file IDs)" backend — host-level CIFS moves may break file-ID invariant |
| 6 | Build 3-year retention pruning cron | DEFERRED | Weekly `find /Archive -type f -mtime +1095 -delete` then `occ files:scan pavon/Archive` |

### Concrete next session checklist

1. Decide approach for pavon versioning: A (disable globally), C (aggressive 30-day migration), or D (`versions:cleanup pavon` daily cron) — D is what user proposed
2. If D: run one-time `occ versions:cleanup pavon` (reclaims 30 GB) + `occ trashbin:cleanup pavon`, then schedule `0 3 * * *` cron
3. Decide migration cutoff (90 days is current target; could go shorter for capacity reasons since `/owncloud` is 74% full)
4. Build migration script — open question whether to use OwnCloud API or host-level CIFS mount + `mv`
5. Build retention pruning cron on Uranus or via CIFS mount on OwnCloud VM
6. Reconcile the 1Password OwnCloud password (currently has stale value `Paper123!@#-unifi!`; should be `r3tr0gradE99!!` per SOPS)
7. Optionally clean up: `occ group:delete versioning_users` (created but unused this session)

---

## Reference

### File paths

- OwnCloud install root: `/var/www/owncloud`
- OwnCloud occ command: `sudo -u apache php /var/www/owncloud/occ ...`
- OwnCloud data dir: `/owncloud` (NFS to Jupiter)
- Pavon's home files: `/owncloud/pavon/files/`
- Pavon's versions junk: `/owncloud/pavon/files_versions/` (30 GB, 1,326 files)
- Apache crontab: `/var/spool/cron/apache`
- Crontab backup: `/root/apache-crontab.backup-20260428-pre-flock`
- Cron lock file: `/tmp/oc-cron.lock` (used by flock wrapper)

### Useful occ commands for next session

```bash
# Per-user version cleanup (deletes BOTH disk files and filecache rows)
sudo -u apache php /var/www/owncloud/occ versions:cleanup pavon
sudo -u apache php /var/www/owncloud/occ trashbin:cleanup pavon

# Trigger a scan after filesystem-level changes
sudo -u apache php /var/www/owncloud/occ files:scan pavon
sudo -u apache php /var/www/owncloud/occ files:scan pavon/Archive

# Per-user expire (uses retention policy)
sudo -u apache php /var/www/owncloud/occ versions:expire pavon
sudo -u apache php /var/www/owncloud/occ trashbin:expire pavon

# External storage management
sudo -u apache php /var/www/owncloud/occ files_external:list
sudo -u apache php /var/www/owncloud/occ files_external:option <id> filesystem_check_changes 0
```

### Local files created this session

All under `c:/Users/guru/ClaudeTools/temp/` (small bash scripts uploaded via pscp to /tmp on OwnCloud VM):

- `owncloud-investigate.sh` — initial pavon storage walk
- `owncloud-versioning-check.sh` — schema/state check
- `owncloud-groups-check.sh` — group enumeration
- `owncloud-pavon-groups.sh` — per-user group mapping
- `owncloud-versioning-restrict.sh` — failed group-restrict attempt
- `occ-versions-help.sh` — discover available occ subcommands

These are scratch scripts; no need to preserve.

### MariaDB on OwnCloud VM (cheat sheet)

- Version: MariaDB 10.5.29
- Local CLI: `mysql owncloud --skip-column-names <<<'SQL...'` works as root via socket auth
- `mysql -e 'SQL'` does NOT work via plink heredoc; PowerShell mangles quoting. Use heredoc (`<<<` or `<<EOF`) instead

---

## Notes for Howard / Future Mike

- **Don't kill the OwnCloud VM** while this OwnCloud + GeoVision NVR architecture is in place — pavon's NVRs are dependent on it being reachable to either upload new footage OR rehydrate placeholders if footage is requested
- **The flock wrapper is the only thing keeping the cron stable** right now. Any future change to `/var/spool/cron/apache` should preserve `flock -n /tmp/oc-cron.lock`
- **The bad query is a property of OwnCloud's `files_versions` background job + camera filename pattern + filecache schema** — not specific to anything we did. If we ever upgrade OwnCloud, retest under load before declaring victory
- **Storage 6 (`/Archive`) is already configured correctly** — don't recreate it. Just use it
- **30 GB cleanup waiting** — `occ versions:cleanup pavon` will reclaim it instantly when you're ready

---

## Status at session end

- OwnCloud VM load: dropping past **5.95** (1-min) from a peak of 80, recovering toward normal
- MariaDB CPU: oscillating 0-140% (normal background)
- Stale cron processes: **0**
- New cron via flock: armed; first protected tick will fire within 15 minutes of the change
- BirthBiologic SPMT (the original ticket): unchanged on the Windows VM side; the user stopped SPMT manually during diagnosis. Decision pending whether to resume SPMT or switch to OneDrive client (separate question discussed mid-session)
