# 22nd St office "slow internet" investigation (dead PoE switch + AP, Winter fix) + Factorio mod design

## User
- **User:** Mike Swanson (mike)
- **Machine:** GURU-5070
- **Role:** admin

## Session Summary

Two threads. **(1) ACG 22nd St office network troubleshooting** — reported "internet slower than
normal," with Winter's phone showing Wi-Fi "no internet" and her wired PC getting partial page
loads. Worked it top-down. The office pfSense (172.16.0.1, SSH :2248, pfSense 2.8.1) and its single
Cox fiber WAN tested fully healthy: 0% loss, 14 ms, ~570 Mbps down / ~385 up single-stream, 1G
full-duplex WAN NIC with 0 errors, full-1500 MTU passes, DNS reliable, CPU idle, DHCP pool
(172.16.1.1-254, 2 h leases, ~28 active devices) nowhere near exhaustion. Confirmed it is a flat
/22 (172.16.0.0/22) with no VLANs — the `192.168.206.x` seen on Winter's Wi-Fi was stale junk, not
a real guest VLAN.

Pivoted to Winter's PC (DESKTOP-U303G5J, added to GuruRMM mid-session): wired path was clean (1G
FD, 0 loss, 476 Mbps). Her **partial page loads were caused by a broken Wi-Fi adapter** — APIPA
169.254 (no DHCP) with a **stale static DNS 192.168.206.48** that bled into Windows' multi-adapter
DNS resolution and caused intermittent lookup timeouts. **Disabled the Wi-Fi adapter via RMM**
(she's wired) — DNS now clean, resolved.

Then the office Wi-Fi via the UniFi controller (UOS 172.16.3.29; site "22nd St"). gw-audit showed
disconnected devices; ping + controller stat confirmed the **root cause: the US-8-60W PoE switch
(172.16.1.8) is DOWN**, which took the **U7-Pro AP (172.16.1.34) offline** — office running on 1 of
2 APs → Wi-Fi unusable in that AP's area (Winter's phone). Also found the **USW Pro 24 Fiber Port
24 linked at only 10 Mbps** (1G-capable; far end non-UniFi, no LLDP) = a throttled segment, and the
**USW-Lite-8-PoE (172.16.1.12)** up but unmanaged. **Remote fixes:** re-adopted the .12 switch.
Created Syncro ticket **#32454** (Arizona Computer Guru, cust 15353550) with the full findings +
on-site dispatch note (power-cycle the .8 switch, check the port-24 SFP/fiber). The dead AC switch +
the SFP need on-site hands.

**(2) Factorio mod — design discussion** (personal project). Researched Factorio 2.0 mod anatomy
(Grok live web; Gemini CLI auth is broken on this box). Mike wants a **deterministic, cost-based
quality system** replacing vanilla's RNG quality: a Refinery machine converts N lower-quality items
into 1 higher-quality item (no chance), keeping vanilla's quality tiers + perks untouched, with an
escalating compounding cost (≈3/4/5/6 per tier step). Captured the full design + the 2.0 mod-anatomy
reference under `projects/factorio-quality-mod/` (committed earlier this session, 30841fbf).

## Key Decisions
- **pfSense exonerated by measurement, not assumption** — ran loss/latency/throughput/MTU/DNS/DHCP
  before concluding; ruled out WAN, DHCP exhaustion, and MTU before pivoting downstream.
- **Diagnose from the user's actual machine** — adding Winter's PC to RMM let me prove the wired path
  was clean and isolate the dead Wi-Fi adapter as the partial-load cause.
- **Disabled (not reconfigured) Winter's Wi-Fi adapter** — she's wired; killing the broken adapter
  removes the bad-DNS interference cleanly and reversibly.
- **On-site dispatch via a Syncro ticket** for the physical items (dead AC PoE switch, 10M SFP) —
  can't power-cycle a fully-offline AC switch or reseat an SFP remotely.
- **Factorio mod = full replacement of RNG quality**, keep vanilla tiers/perks, deterministic
  Refinery, escalating compounding cost (Mike, 2026-06-23).

## Problems Encountered
- **MSYS path conversion** mangled `/bin/sh` to a Windows path when passed to ssh.exe (Git-Bash
  gotcha) — fixed with `MSYS_NO_PATHCONV=1` + a non-slash remote command (`sh -s`).
- **Gemini CLI auth broken** (`throwIneligibleOrProjectIdError` / `_doSetupUser`) — empty responses;
  needs interactive `gemini` re-login. Logged to errorlog; fell back to Grok for research.
- **Old mongo shell rejected `else if`** in the uos-mongo query — rewrote with separate `if` blocks.
- **First post-reboot verify (CCroom1New, earlier)** read stale uptime in the reboot grace window —
  re-verified after the box was down/up.
- **VWP-QBS firewall correction (carryover):** logged that the disabled firewall is intentional for
  testing — leave it (don't re-flag).

## Configuration Changes
- **DESKTOP-U303G5J (Winter):** disabled the `Wi-Fi` net adapter via RMM (`Disable-NetAdapter`);
  Ethernet-only, DNS now 172.16.0.1/8.8.8.8/1.1.1.1.
- **UniFi 22nd St:** re-adopted USW-Lite-8-PoE 172.16.1.12 (device-control adopt, rc:ok).
- **Syncro #32454** created (Arizona Computer Guru 15353550) — findings + on-site dispatch comment.
- **New project** `projects/factorio-quality-mod/` — `DESIGN.md` + `factorio-mod-anatomy.md` (committed 30841fbf).
- No pfSense changes (read-only diagnostics only).

## Credentials & Secrets
- None created. Read-only use of: office pfSense SSH key (`C:/Users/guru/.ssh/id_ed25519`, admin@172.16.0.1:2248);
  `infrastructure/uos-server-ssh-key` (uos-mongo), `infrastructure/uos-server-network-api-rw`
  (controller stat read), Syncro mike key, GuruRMM admin. pfSense admin pw vault
  `infrastructure/pfsense-firewall` (unused — key auth worked).

## Infrastructure & Servers
- **Office pfSense** 172.16.0.1 (SSH :2248, web :4433), pfSense 2.8.1, single Cox fiber WAN
  (igc0 98.181.90.163/31, gw .162), flat LAN igc2 172.16.0.0/22, DHCP kea 172.16.1.1-254 (2 h).
  Tailscale 100.119.153.74. Stale dead `OptGW` (184.187.220.90) still monitored (false alarms).
- **UniFi UOS** 172.16.3.29 (controller :11443; mongo via uos-server-ssh-key). Site "22nd St" =
  `5f493a90c9e77c010bbb134c` (short `1p7jvx8r`).
  - U7-Pro AP `172.16.1.34` (mac 28:70:4e:d5:36:cf) — **DOWN**.
  - UAPA6A9 AP `172.16.1.63` — up.
  - US-8-60W switch `172.16.1.8` (mac f4:e2:c6:e4:3e:dd) — **DOWN** (AC-powered).
  - USW Pro 24 `172.16.1.11` — up; **Fiber Port 24 @ 10M** (10G SFP+ 25/26 unused).
  - USW-Lite-8-PoE `172.16.1.12` — re-adopted.
  - DMarc USMINI `172.16.1.15` — up.
- **Winter PC** DESKTOP-U303G5J 172.16.1.158, GuruRMM agent `52c90de1-6d58-4654-a6ed-b779b8ad93fc`.
- Syncro: Arizona Computer Guru cust **15353550** (7437 E 22nd St).

## Commands & Outputs
- pfSense diag over SSH (key auth): `MSYS_NO_PATHCONV=1 ssh ... -p 2248 admin@172.16.0.1 sh -s <<EOF`
  ping/netstat/pfctl/ifconfig/curl speedtest. Down ~71.2 MB/s (50MB Cloudflare), up ~48 MB/s.
- UOS mongo: `echo 'db.device.find({site_id:...})...' | bash .claude/scripts/uos-mongo.sh`.
- Controller stat: POST $BASE/api/auth/login → GET /proxy/network/api/s/1p7jvx8r/stat/device (RW cred).
- unifi-wifi: `gw-audit.sh "22nd St"`, `switch-audit.sh "22nd St" --all-ports`, `device-control.sh "22nd St" adopt <mac> --apply`.
- Research: `ask-grok.sh text --prompt-file ...` (Gemini failed auth).

## Pending / Incomplete Tasks
- **#32454 on-site (HIGH):** power-cycle/check US-8-60W (172.16.1.8) → restores U7-Pro AP + office Wi-Fi.
- **#32454 on-site:** inspect/reseat Fiber Port 24 SFP+fiber on USW Pro 24 (10M→1G); trace what it feeds.
- Optional: confirm the "USW Pro Max 16 PoE" (controller-offline/never-onboarded) is actually in service.
- VWP-QBS firewall stays OFF intentionally (do not re-enable) until VWP testing done.
- **Gemini CLI:** interactive re-login needed on GURU-5070 to restore that research path.
- Office pfSense cruft: remove the dead `OptGW` gateway (stops false down-alarms) — needs go (single point of failure).
- **Factorio mod:** next = feasibility check (deterministic quality-up recipe; neutralize quality-module RNG). See `projects/factorio-quality-mod/DESIGN.md`.

## Reference Information
- Ticket #32454 (id 112996018) https://computerguru.syncromsp.com/tickets/112996018 (comment 420428386).
- #dev-alerts/#bot-alerts posts this session: Winter Wi-Fi, ticket #32454.
- Factorio project: `projects/factorio-quality-mod/{DESIGN.md,factorio-mod-anatomy.md}` (commit 30841fbf).
- Related earlier logs: clients/valleywide/.../2026-06-23-mike-vwp-smb1-orders-xp-g-drive.md; session-logs/2026-06/2026-06-23-mike-vwp-qbs-firewall-ccroom1new-uac.md.
