Compare commits
5 Commits
c98692e424
...
7be8f454e0
| Author | SHA1 | Date | |
|---|---|---|---|
| 7be8f454e0 | |||
| 761bae5d01 | |||
| a062a825ea | |||
| b1862800a1 | |||
| 5e2325507f |
@@ -61,6 +61,7 @@ Bringing GC to parity with GuruRMM's release engineering. Full plan: [SPEC-001](
|
||||
- [x] Support-code (attended) and persistent (unattended) agent modes
|
||||
- [x] Protobuf-over-WSS transport, Zstd frame compression
|
||||
- [~] React/TS web viewer (`dashboard/src/components/RemoteViewer.tsx`) — embeddable session viewer
|
||||
- [ ] **Headless Linux mode (direct TTY access)** — P2 — Terminal-based remote access for Linux servers without GUI. PTY spawn (`openpty`), xterm.js web viewer, full ANSI/VT100 support. Enables server management, container debugging, emergency recovery via GuruConnect dashboard with audit logging. SSH replacement with centralized auth. ([SPEC-012](specs/SPEC-012-headless-linux-tty.md))
|
||||
- [ ] Multi-monitor switching — P2
|
||||
- [ ] File transfer — P3 (out of scope for native-remote-control v1)
|
||||
- [ ] Session recording — P3 (out of scope for native-remote-control v1)
|
||||
@@ -93,5 +94,6 @@ Bringing GC to parity with GuruRMM's release engineering. Full plan: [SPEC-001](
|
||||
|
||||
## Future Considerations
|
||||
|
||||
- [ ] macOS / Linux remote-control agents — P3
|
||||
- [ ] **Cross-platform agent support (macOS and Linux)** — P2 — Enable remote control beyond Windows with native agents for macOS 12+ and Ubuntu 22.04+ LTS. Platform abstraction layer for capture/input/encoding, VideoToolbox (macOS) and VA-API (Linux) H.264 encoding, .app/.deb/.rpm packaging. Unblocks multi-platform MSP adoption. ([SPEC-010](specs/SPEC-010-cross-platform-agents.md))
|
||||
- [ ] **Mobile agent support (iOS and Android as remote control targets)** — P3 — Native mobile apps for supervised support sessions. iOS: ReplayKit screen sharing (view-only, no input injection due to sandboxing). Android: MediaProjection screen capture + Accessibility Service input injection. Support-code authentication, push notifications (APNs/FCM), consent-first model. Foreground-only operation (OS limitation). Distinct from GuruRMM SPEC-017 (MDM/management). ([SPEC-011](specs/SPEC-011-mobile-agents.md))
|
||||
- [ ] Auto-update for the agent — P3
|
||||
|
||||
358
docs/specs/SPEC-010-cross-platform-agents.md
Normal file
358
docs/specs/SPEC-010-cross-platform-agents.md
Normal file
@@ -0,0 +1,358 @@
|
||||
# SPEC-010: Cross-Platform Agent Support (macOS and Linux)
|
||||
|
||||
**Status:** Proposed
|
||||
**Priority:** P2
|
||||
**Requested By:** Mike Swanson (2026-05-30)
|
||||
**Estimated Effort:** X-Large
|
||||
|
||||
## Overview
|
||||
|
||||
Expand GuruConnect's agent support beyond Windows to include macOS and Linux platforms, enabling remote control and support across the full spectrum of enterprise and home computing environments. This cross-platform expansion addresses a critical market gap—ScreenConnect, Splashtop, and AnyDesk all support macOS/Linux, and GuruConnect's Windows-only limitation blocks adoption by multi-platform organizations. The primary technical challenge is abstracting platform-specific screen capture, input injection, and video encoding while maintaining the core protobuf-over-WSS transport and session model. Success criteria: feature parity with the Windows agent (capture, input, tray presence, persistent/support-code modes, H.264 encoding) on macOS 12+ and Ubuntu 22.04+ LTS.
|
||||
|
||||
## Scope
|
||||
|
||||
### Included in v1
|
||||
|
||||
**macOS Agent (Priority 1):**
|
||||
- Screen capture via ScreenCaptureKit API (macOS 13+) with AVFoundation fallback (macOS 12)
|
||||
- Input injection via CGEvent
|
||||
- H.264 encoding via VideoToolbox
|
||||
- Menu bar icon (NSStatusItem) with status and exit option
|
||||
- `guruconnect://` protocol handler registration
|
||||
- .app bundle packaging and code signing
|
||||
- Screen Recording permission prompt and validation
|
||||
- Universal binary (x86_64 + arm64)
|
||||
|
||||
**Linux Agent (Priority 2):**
|
||||
- Screen capture via X11 (XShm) with Wayland pipewire fallback detection
|
||||
- Input injection via XTest (X11) or uinput (Wayland)
|
||||
- H.264 encoding via VA-API (hardware) or software fallback (libx264)
|
||||
- System tray icon via StatusNotifier/AppIndicator
|
||||
- `guruconnect://` protocol handler (.desktop file)
|
||||
- .deb packaging (Ubuntu/Debian) and .rpm (Fedora/RHEL)
|
||||
- x86_64 binary
|
||||
|
||||
**Shared Cross-Platform:**
|
||||
- Unified agent codebase with platform-specific modules behind traits
|
||||
- Same protobuf protocol (no wire format changes)
|
||||
- Same persistent/support-code authentication modes
|
||||
- Same AgentStatus metadata reporting (OS, architecture, uptime, logged-on user)
|
||||
- Viewer compatibility: existing native Windows viewer and web viewer work unchanged
|
||||
|
||||
### Explicitly out of scope
|
||||
|
||||
- **FreeBSD/BSD support** — defer; assess demand post-launch
|
||||
- **Wayland-native screen capture** — v1 uses X11 compatibility layer; native Wayland portal support is Phase 2
|
||||
- **Multi-monitor switching on macOS/Linux** — already deferred on Windows (P2 roadmap item); cross-platform parity follows
|
||||
- **macOS < 12** — 12.x is the minimum (released Oct 2021, 95%+ adoption)
|
||||
- **Linux distros outside Ubuntu 22.04+ / Debian 11+ / Fedora 38+ / RHEL 9+** — official support limited; community builds acceptable
|
||||
|
||||
## Architecture
|
||||
|
||||
### Agent Refactor: Platform Abstraction Layer
|
||||
|
||||
Current Windows agent (`agent/src/`) tightly couples platform APIs. Refactor into:
|
||||
|
||||
```
|
||||
agent/src/
|
||||
├── main.rs # CLI entry + platform dispatch
|
||||
├── session/ # Session logic (platform-agnostic)
|
||||
├── transport/ # WebSocket (unchanged)
|
||||
├── platform/ # NEW: trait definitions
|
||||
│ ├── mod.rs # PlatformCapture, PlatformInput, PlatformTray traits
|
||||
│ ├── windows/ # Windows impl (existing code refactored)
|
||||
│ │ ├── capture.rs # DXGI + GDI behind PlatformCapture
|
||||
│ │ ├── input.rs # Win32 SendInput behind PlatformInput
|
||||
│ │ ├── encoder.rs # Media Foundation H.264
|
||||
│ │ └── tray.rs # Win32 shell tray
|
||||
│ ├── macos/ # NEW: macOS impl
|
||||
│ │ ├── capture.rs # ScreenCaptureKit/AVFoundation
|
||||
│ │ ├── input.rs # CGEvent
|
||||
│ │ ├── encoder.rs # VideoToolbox H.264
|
||||
│ │ └── tray.rs # NSStatusItem (Objective-C via objc2 crate)
|
||||
│ └── linux/ # NEW: Linux impl
|
||||
│ ├── capture.rs # X11 XShm / pipewire detection
|
||||
│ ├── input.rs # XTest / uinput
|
||||
│ ├── encoder.rs # VA-API / libx264
|
||||
│ └── tray.rs # StatusNotifier
|
||||
├── encoder/ # Color conversion, raw fallback (platform-agnostic)
|
||||
├── viewer/ # Native viewer (winit cross-platform, unchanged)
|
||||
├── config.rs # Config (unchanged)
|
||||
└── install.rs # Installation (platform-specific #[cfg] blocks)
|
||||
```
|
||||
|
||||
**Key traits:**
|
||||
|
||||
```rust
|
||||
// agent/src/platform/mod.rs
|
||||
pub trait PlatformCapture: Send {
|
||||
fn capture_frame(&mut self) -> Result<CapturedFrame>;
|
||||
fn get_dimensions(&self) -> (u32, u32);
|
||||
}
|
||||
|
||||
pub trait PlatformInput: Send {
|
||||
fn inject_mouse(&mut self, event: MouseEvent) -> Result<()>;
|
||||
fn inject_keyboard(&mut self, event: KeyboardEvent) -> Result<()>;
|
||||
}
|
||||
|
||||
pub trait PlatformEncoder: Send {
|
||||
fn encode(&mut self, frame: &[u8], width: u32, height: u32) -> Result<Vec<u8>>;
|
||||
fn set_bitrate(&mut self, kbps: u32);
|
||||
}
|
||||
|
||||
pub trait PlatformTray: Send {
|
||||
fn show(&mut self, status: &str) -> Result<()>;
|
||||
fn poll_events(&mut self) -> Vec<TrayEvent>;
|
||||
}
|
||||
```
|
||||
|
||||
### Protobuf Changes
|
||||
|
||||
**None.** The existing `AgentStatus` message already carries OS/arch metadata; viewer consumes frames opaquely. Protocol remains unchanged.
|
||||
|
||||
### Database Schema
|
||||
|
||||
**No migration required.** The `connect_machines` table's `os` and `architecture` fields (part of SPEC-003 inventory) already accommodate macOS/Linux values.
|
||||
|
||||
### Agent Build & Packaging
|
||||
|
||||
**Cargo.toml adjustments:**
|
||||
|
||||
```toml
|
||||
[target.'cfg(target_os = "macos")'.dependencies]
|
||||
core-graphics = "0.23"
|
||||
core-foundation = "0.9"
|
||||
objc2 = "0.5"
|
||||
objc2-foundation = "0.2"
|
||||
objc2-app-kit = "0.2"
|
||||
security-framework = "2.9"
|
||||
|
||||
[target.'cfg(target_os = "linux")'.dependencies]
|
||||
x11 = { version = "2.21", features = ["xlib", "xtest", "xrandr"] }
|
||||
libva = "0.4"
|
||||
```
|
||||
|
||||
**macOS packaging:**
|
||||
|
||||
- `.app` bundle structure: `GuruConnect.app/Contents/MacOS/guruconnect`, `Info.plist` with protocol handler
|
||||
- Code signing: `codesign --sign "Developer ID Application: Arizona Computer Guru LLC"`
|
||||
- Notarization via `notarytool` (Apple requirement for macOS 10.15+)
|
||||
- Universal binary: `cargo build --release --target x86_64-apple-darwin && cargo build --release --target aarch64-apple-darwin && lipo -create ...`
|
||||
|
||||
**Linux packaging:**
|
||||
|
||||
- `.deb` via `cargo-deb`:
|
||||
```toml
|
||||
[package.metadata.deb]
|
||||
maintainer = "AZ Computer Guru <support@azcomputerguru.com>"
|
||||
depends = "libx11-6, libxext6, libxtst6"
|
||||
section = "net"
|
||||
assets = [
|
||||
["target/release/guruconnect", "usr/bin/", "755"],
|
||||
["deploy/linux/guruconnect.desktop", "usr/share/applications/", "644"],
|
||||
]
|
||||
```
|
||||
- `.rpm` via `cargo-generate-rpm`
|
||||
|
||||
### Distribution
|
||||
|
||||
- **Server downloads:** `/downloads/guruconnect-macos.dmg`, `/downloads/guruconnect-linux.deb`, `/downloads/guruconnect-linux.rpm`
|
||||
- **Dashboard detection:** browser User-Agent → suggest correct download
|
||||
- **Auto-update (out of scope for v1):** defer cross-platform updater to Phase 2
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Files to Create
|
||||
|
||||
**macOS:**
|
||||
- `agent/src/platform/macos/capture.rs` — ScreenCaptureKit stream + AVFoundation fallback
|
||||
- `agent/src/platform/macos/input.rs` — CGEventCreateMouseEvent, CGEventCreateKeyboardEvent, CGEventPost
|
||||
- `agent/src/platform/macos/encoder.rs` — VTCompressionSession (VideoToolbox)
|
||||
- `agent/src/platform/macos/tray.rs` — NSStatusBar + NSMenu via objc2
|
||||
- `agent/Info.plist.template` — bundle metadata, protocol handler (`CFBundleURLSchemes`)
|
||||
|
||||
**Linux:**
|
||||
- `agent/src/platform/linux/capture.rs` — X11Display::open, XShmGetImage; pipewire detection fallback (log warning)
|
||||
- `agent/src/platform/linux/input.rs` — XTestFakeMotionEvent, XTestFakeButtonEvent, XTestFakeKeyEvent
|
||||
- `agent/src/platform/linux/encoder.rs` — libva VAConfigAttrib + VABufferType; software fallback via openh264
|
||||
- `agent/src/platform/linux/tray.rs` — dbus org.kde.StatusNotifierItem protocol
|
||||
- `deploy/linux/guruconnect.desktop` — .desktop file with `MimeType=x-scheme-handler/guruconnect`
|
||||
|
||||
**Shared:**
|
||||
- `agent/src/platform/mod.rs` — trait definitions + platform factory `fn create_platform() -> Box<dyn Platform>`
|
||||
- Refactor `agent/src/capture/{dxgi,gdi}.rs` → `agent/src/platform/windows/capture.rs`
|
||||
- Refactor `agent/src/input/{mouse,keyboard}.rs` → `agent/src/platform/windows/input.rs`
|
||||
- Refactor `agent/src/encoder/h264.rs` → `agent/src/platform/windows/encoder.rs` (Media Foundation)
|
||||
|
||||
### Key Logic
|
||||
|
||||
**macOS screen capture (ScreenCaptureKit, macOS 13+):**
|
||||
|
||||
```rust
|
||||
// agent/src/platform/macos/capture.rs
|
||||
use core_graphics::display::CGDisplay;
|
||||
use objc2::rc::Id;
|
||||
use objc2_foundation::NSArray;
|
||||
use objc2_screen_capture_kit::{SCStreamConfiguration, SCStream, SCContentFilter, SCStreamOutputType};
|
||||
|
||||
pub struct MacOSCapture {
|
||||
stream: Id<SCStream>,
|
||||
latest_frame: Arc<Mutex<Option<Vec<u8>>>>,
|
||||
}
|
||||
|
||||
impl PlatformCapture for MacOSCapture {
|
||||
fn capture_frame(&mut self) -> Result<CapturedFrame> {
|
||||
// ScreenCaptureKit delivers frames to delegate callback
|
||||
let frame = self.latest_frame.lock().unwrap().clone()
|
||||
.ok_or(anyhow!("No frame available"))?;
|
||||
Ok(CapturedFrame { data: frame, width: self.width, height: self.height })
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**macOS input injection:**
|
||||
|
||||
```rust
|
||||
// agent/src/platform/macos/input.rs
|
||||
use core_graphics::event::{CGEvent, CGEventType, CGMouseButton, EventField};
|
||||
|
||||
impl PlatformInput for MacOSInput {
|
||||
fn inject_mouse(&mut self, event: MouseEvent) -> Result<()> {
|
||||
let cg_event = match event.button {
|
||||
MouseButton::Left => CGEvent::new_mouse_event(
|
||||
None, CGEventType::LeftMouseDown,
|
||||
CGPoint::new(event.x as f64, event.y as f64),
|
||||
CGMouseButton::Left
|
||||
)?,
|
||||
// ...
|
||||
};
|
||||
cg_event.post(CGEventTapLocation::HID);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Linux X11 capture:**
|
||||
|
||||
```rust
|
||||
// agent/src/platform/linux/capture.rs
|
||||
use x11::xlib::{XOpenDisplay, XDefaultRootWindow, XGetImage, ZPixmap};
|
||||
use x11::xshm::{XShmAttach, XShmGetImage};
|
||||
|
||||
pub struct LinuxCapture {
|
||||
display: *mut Display,
|
||||
root: Window,
|
||||
shm_info: XShmSegmentInfo,
|
||||
}
|
||||
|
||||
impl PlatformCapture for LinuxCapture {
|
||||
fn capture_frame(&mut self) -> Result<CapturedFrame> {
|
||||
let img = unsafe { XShmGetImage(self.display, self.root, ...) };
|
||||
// Convert XImage BGRA → frame buffer
|
||||
Ok(CapturedFrame { data: converted, width, height })
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Permission handling (macOS Screen Recording):**
|
||||
|
||||
```rust
|
||||
// agent/src/platform/macos/permissions.rs
|
||||
use objc2_av_foundation::AVCaptureDevice;
|
||||
|
||||
pub fn check_screen_recording_permission() -> bool {
|
||||
// macOS 10.15+ requires explicit Screen Recording permission
|
||||
// Trigger prompt on first run; subsequent checks use TCC database status
|
||||
AVCaptureDevice::authorizationStatusForMediaType(AVMediaTypeVideo) == AVAuthorizationStatusAuthorized
|
||||
}
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### macOS Security
|
||||
|
||||
- **Screen Recording permission:** User must grant permission in System Settings → Privacy & Security → Screen Recording. Agent detects denial and logs instructions.
|
||||
- **Gatekeeper:** Code signing + notarization required to avoid "unidentified developer" block.
|
||||
- **Hardened runtime:** Enable hardened runtime entitlements (`com.apple.security.cs.allow-unsigned-executable-memory` for JIT, if needed).
|
||||
- **Input injection:** No additional permission for CGEvent (runs as the logged-on user).
|
||||
|
||||
### Linux Security
|
||||
|
||||
- **X11 security:** XTest extension enabled by default; uinput requires `/dev/uinput` write access (typically granted to `input` group).
|
||||
- **Wayland sandboxing:** v1 runs under XWayland (X11 compatibility); native Wayland requires PipeWire portal (user consent per-session).
|
||||
- **AppArmor/SELinux:** Agent may need profile exemptions for screen capture on hardened distros.
|
||||
|
||||
### Authentication
|
||||
|
||||
**No protocol changes.** macOS/Linux agents authenticate with the same support-code or persistent agent key (`AGENT_API_KEY` or per-agent `cak_*` key from SPEC-004). Relay server validates identically.
|
||||
|
||||
### Audit Events
|
||||
|
||||
Existing `events` table logs agent connections with `os` and `architecture` fields. No schema change needed.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
|
||||
- Mock platform traits for capture/input/encoder modules
|
||||
- Test frame conversion (BGRA → RGB, stride alignment)
|
||||
- Test protobuf message serialization (unchanged, but validate on new platforms)
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- **macOS test rig:** macOS 13 VM (UTM or Parallels), run agent, verify viewer connects and displays screen
|
||||
- **Linux test rig:** Ubuntu 22.04 LXC container (X11 forwarded), verify capture + input
|
||||
- **Cross-platform viewer:** existing Windows native viewer connects to macOS/Linux agents
|
||||
|
||||
### Manual Testing Scenarios
|
||||
|
||||
1. **macOS install:** Download `.app`, drag to Applications, launch, grant Screen Recording permission, enter support code, verify viewer connects.
|
||||
2. **Linux install:** `sudo dpkg -i guruconnect.deb`, run `guruconnect --support-code ABC123`, verify tray icon, test remote control.
|
||||
3. **Encoding quality:** Compare H.264 output from VideoToolbox (macOS), VA-API (Linux), Media Foundation (Windows) — validate frame rate and bitrate parity.
|
||||
4. **Input fidelity:** Test full keyboard (including modifiers, function keys) and mouse (click, drag, scroll) on all three platforms.
|
||||
5. **Protocol handler:** Click `guruconnect://connect?session=...` link, verify viewer launches with session ID pre-filled.
|
||||
|
||||
### CI/CD Additions
|
||||
|
||||
- **macOS build:** Gitea Actions runner on macOS (M1 or x86_64), build universal binary, sign, notarize
|
||||
- **Linux build:** Existing `ubuntu-latest` runner, build `.deb` and `.rpm` via cargo-deb/cargo-generate-rpm
|
||||
- **Artifact upload:** Upload binaries to `server/static/downloads/` on release tag
|
||||
|
||||
## Effort Estimate & Dependencies
|
||||
|
||||
**Size:** X-Large (12-16 weeks, 1 developer)
|
||||
|
||||
**Breakdown:**
|
||||
- Platform abstraction refactor (Windows code): 2 weeks
|
||||
- macOS implementation (capture, input, encoder, tray): 4 weeks
|
||||
- Linux implementation (X11 capture, input, VA-API encoder, tray): 3 weeks
|
||||
- Packaging and code signing (macOS notarization, .deb/.rpm): 2 weeks
|
||||
- Testing, documentation, CI/CD integration: 2 weeks
|
||||
- Buffer for platform-specific edge cases: 1-3 weeks
|
||||
|
||||
**Dependencies:**
|
||||
- **SPEC-002 v2 Phase 1 completion** — per-agent keys and secure session core must be stable (already shipped)
|
||||
- **SPEC-004 deterministic machine identity** — macOS/Linux need stable `machine_uid` derivation (macOS: `IOPlatformUUID`, Linux: `/etc/machine-id`)
|
||||
- **Code signing infrastructure:** Apple Developer Program account + Azure Trusted Signing already in place (for Windows); reuse for macOS
|
||||
- **Test infrastructure:** macOS VM or bare-metal test host, Linux LXC container with X11
|
||||
|
||||
**Unblocks:**
|
||||
- Multi-platform MSP adoption (organizations with mixed Windows/macOS/Linux fleets)
|
||||
- Education/developer market (high macOS/Linux penetration)
|
||||
- SPEC-003 machine inventory (macOS/Linux populate same fields)
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Wayland native support timeline?** — v1 uses X11 compatibility (XWayland); when do we commit to PipeWire portal? Defer to Phase 2 pending Wayland adoption metrics.
|
||||
2. **ARM Linux (Raspberry Pi, ARM servers)?** — Ubuntu 22.04 arm64 is viable; add as a build target? Defer to community request.
|
||||
3. **macOS Ventura (13.0) vs. Monterey (12.0) adoption split?** — ScreenCaptureKit (13.0+) is preferred; fallback to AVFoundation for 12.x. Monitor 12.x usage after 6 months; deprecate if <5%.
|
||||
4. **Linux H.264 licensing?** — VA-API uses hardware codecs (no license issue); software fallback via openh264 (Cisco BSD license, royalty-free). Confirm distro packaging allows bundling.
|
||||
5. **System tray on headless Linux servers?** — Agent can run headless (tray is optional); `--no-tray` flag for server deployments.
|
||||
|
||||
---
|
||||
|
||||
**Cross-references:**
|
||||
- ADR-001: GuruConnect is a standalone product (no RMM coupling)
|
||||
- SPEC-002: v2 modernization architecture (agent key model)
|
||||
- SPEC-003: Machine inventory (OS/arch fields)
|
||||
- SPEC-004: Stable machine identity (macOS IOPlatformUUID, Linux /etc/machine-id)
|
||||
482
docs/specs/SPEC-011-mobile-agents.md
Normal file
482
docs/specs/SPEC-011-mobile-agents.md
Normal file
@@ -0,0 +1,482 @@
|
||||
# SPEC-011: Mobile Agent Support (iOS and Android as Remote Control Targets)
|
||||
|
||||
**Status:** Proposed
|
||||
**Priority:** P3
|
||||
**Requested By:** Mike Swanson (2026-05-30)
|
||||
**Estimated Effort:** X-Large (16-20 weeks, requires mobile development expertise)
|
||||
|
||||
## Overview
|
||||
|
||||
Enable remote viewing and control of iOS/Android mobile devices by building native GuruConnect agent apps for the App Store and Google Play. Unlike desktop agents that run persistently in the background, mobile agents operate within OS sandbox constraints: they require user consent to share the screen, must remain in the foreground during sessions, and (on iOS) cannot inject input at all. The primary use case is **supervised support sessions**—a user opens the app, shares their screen with a technician, and the technician can see the screen (both platforms) and remotely control it (Android only). This is fundamentally different from desktop remote control; it's an attended, consent-driven support tool constrained by mobile OS security models.
|
||||
|
||||
**Use Cases:**
|
||||
- Support technician walks a customer through app setup while viewing their mobile screen in real-time
|
||||
- Android device troubleshooting with remote control (tap, swipe, type) via Accessibility Service
|
||||
- iOS device screen sharing for demonstration or guided support (view-only, user retains control)
|
||||
|
||||
**Success Criteria:**
|
||||
- iOS app (iOS 14+) shares screen via ReplayKit with user consent; viewer sees live screen; no input injection
|
||||
- Android app (Android 10+) shares screen via MediaProjection and accepts remote input via Accessibility Service
|
||||
- Both apps connect using same protobuf-over-WSS protocol and support-code authentication as desktop agents
|
||||
- Push notifications wake the app when a support session is requested
|
||||
|
||||
## Scope
|
||||
|
||||
### Included in v1
|
||||
|
||||
**iOS Agent (View-Only):**
|
||||
- Native Swift/SwiftUI app targeting iOS 14+ and iPadOS 14+
|
||||
- Screen capture via ReplayKit 2 (`RPScreenRecorder`, `RPBroadcastSampleHandler`)
|
||||
- H.264 encoding via VideoToolbox
|
||||
- User consent required: "Start Broadcast" button triggers system permission prompt
|
||||
- Foreground-only operation (app must remain visible during session)
|
||||
- Support-code authentication (6-digit code entry)
|
||||
- Push notification (APNs) to alert user of incoming support request
|
||||
- **NO input injection** (iOS sandboxing prevents third-party input; user controls their own device)
|
||||
- Displays "Session Active" banner with duration and "Stop Sharing" button
|
||||
|
||||
**Android Agent (View + Control):**
|
||||
- Native Kotlin/Jetpack Compose app targeting Android 10+ (API 29+)
|
||||
- Screen capture via MediaProjection API (requires user consent per session)
|
||||
- H.264 encoding via MediaCodec
|
||||
- Input injection via Accessibility Service (user must enable in Settings → Accessibility)
|
||||
- Foreground service with persistent notification during session
|
||||
- Support-code authentication (6-digit code entry)
|
||||
- Push notification (FCM) to alert user of incoming support request
|
||||
- Displays ongoing notification: "GuruConnect session active - Tap to stop"
|
||||
|
||||
**Shared Cross-Platform:**
|
||||
- Same protobuf protocol (`AgentStatus`, `FrameData`, `InputEvent`) as desktop agents
|
||||
- Support-code-only authentication (persistent agent mode deferred to Phase 2)
|
||||
- Relay server unchanged (mobile agents are just another platform)
|
||||
- Dashboard shows mobile devices with OS icon (iOS/Android) and "Mobile" badge
|
||||
- Existing native/web viewers display mobile screens without modification
|
||||
|
||||
### Explicitly out of scope
|
||||
|
||||
- **Persistent/unattended agent mode** — v1 is attended-only (user must open the app and consent)
|
||||
- **iOS input injection** — technically impossible without jailbreak or Apple Private APIs (violates App Store guidelines)
|
||||
- **Background screen capture** — both iOS and Android require the app to be foreground during capture
|
||||
- **File transfer** — defer to Phase 2 (not in desktop agents yet per roadmap)
|
||||
- **Chat** — defer to Phase 2 (desktop agents have it, but deprioritized for mobile v1)
|
||||
- **Multi-device support in single app** — one mobile device = one agent instance
|
||||
- **Tablet-optimized UI** — v1 UI is phone-first; iPad/Android tablet use same layout
|
||||
|
||||
## Architecture
|
||||
|
||||
### iOS App Structure
|
||||
|
||||
```
|
||||
GuruConnectMobile-iOS/
|
||||
├── App/
|
||||
│ ├── GuruConnectApp.swift # SwiftUI app entry
|
||||
│ ├── ContentView.swift # Main UI (support code entry, status)
|
||||
│ ├── SessionView.swift # Active session UI (duration, stop button)
|
||||
│ └── Info.plist # Capabilities, permissions
|
||||
├── Broadcast/ # ReplayKit broadcast extension
|
||||
│ ├── SampleHandler.swift # RPBroadcastSampleHandler subclass
|
||||
│ ├── VideoEncoder.swift # VideoToolbox H.264 encoding
|
||||
│ └── Info.plist # Extension config
|
||||
├── Shared/
|
||||
│ ├── Protocol/ # Protobuf messages (Swift codegen)
|
||||
│ ├── Transport/ # WebSocket client (Starscream)
|
||||
│ ├── Authentication.swift # Support code validation
|
||||
│ └── PushNotifications.swift # APNs registration + handling
|
||||
└── GuruConnectMobile.xcodeproj
|
||||
```
|
||||
|
||||
**ReplayKit architecture:**
|
||||
- Main app registers broadcast extension via `RPSystemBroadcastPickerView`
|
||||
- User taps "Start Broadcast" → system shows app picker → user selects GuruConnect
|
||||
- Extension (`SampleHandler`) receives CMSampleBuffers in `processSampleBuffer(_:with:)`
|
||||
- Extension encodes H.264 via VideoToolbox, sends frames to shared App Group container
|
||||
- Main app reads from shared container, sends frames via WebSocket to relay server
|
||||
- App Group required: shared data between app and extension (`group.com.azcomputerguru.guruconnect`)
|
||||
|
||||
### Android App Structure
|
||||
|
||||
```
|
||||
GuruConnectMobile-Android/
|
||||
├── app/src/main/
|
||||
│ ├── java/com/azcomputerguru/guruconnect/
|
||||
│ │ ├── MainActivity.kt # Jetpack Compose UI
|
||||
│ │ ├── SessionActivity.kt # Active session screen
|
||||
│ │ ├── ScreenCaptureService.kt # Foreground service, MediaProjection
|
||||
│ │ ├── InputAccessibilityService.kt # AccessibilityService for input injection
|
||||
│ │ ├── VideoEncoder.kt # MediaCodec H.264 encoding
|
||||
│ │ ├── WebSocketClient.kt # OkHttp WebSocket
|
||||
│ │ ├── ProtobufHandler.kt # Protobuf serialization
|
||||
│ │ └── PushMessagingService.kt # FCM receiver
|
||||
│ ├── res/
|
||||
│ │ ├── layout/ # XML layouts (if not full Compose)
|
||||
│ │ ├── values/strings.xml
|
||||
│ │ └── xml/accessibility_service_config.xml
|
||||
│ └── AndroidManifest.xml # Permissions, services
|
||||
├── proto/ # Protobuf definitions (shared with server)
|
||||
└── build.gradle
|
||||
```
|
||||
|
||||
**MediaProjection architecture:**
|
||||
- User grants MediaProjection permission via `MediaProjectionManager.createScreenCaptureIntent()`
|
||||
- `ScreenCaptureService` (foreground service) creates `VirtualDisplay` → frames to `ImageReader`
|
||||
- `VideoEncoder` encodes frames with `MediaCodec` (H.264)
|
||||
- `WebSocketClient` sends encoded frames to relay server
|
||||
- `InputAccessibilityService` receives `InputEvent` protobuf messages, dispatches `AccessibilityService.dispatchGesture()`
|
||||
|
||||
### Protobuf Changes
|
||||
|
||||
**Minor additions to support mobile-specific metadata:**
|
||||
|
||||
```protobuf
|
||||
// proto/guruconnect.proto
|
||||
|
||||
message AgentStatus {
|
||||
// Existing fields...
|
||||
optional MobileCapabilities mobile_capabilities = 20;
|
||||
}
|
||||
|
||||
message MobileCapabilities {
|
||||
bool supports_input_injection = 1; // false for iOS, true for Android (if Accessibility enabled)
|
||||
bool requires_foreground = 2; // true for both (can't capture in background)
|
||||
bool requires_user_consent = 3; // true for both (MediaProjection/ReplayKit consent)
|
||||
}
|
||||
|
||||
message InputEvent {
|
||||
// Existing MouseEvent/KeyboardEvent...
|
||||
optional TouchEvent touch_event = 3; // NEW: mobile touch events
|
||||
}
|
||||
|
||||
message TouchEvent {
|
||||
enum Action {
|
||||
DOWN = 0;
|
||||
MOVE = 1;
|
||||
UP = 2;
|
||||
}
|
||||
Action action = 1;
|
||||
float x = 2; // normalized 0.0-1.0
|
||||
float y = 3;
|
||||
int32 pointer_id = 4; // for multi-touch
|
||||
}
|
||||
```
|
||||
|
||||
### Database Schema
|
||||
|
||||
**No migration required.** Mobile devices populate existing `connect_machines` table with:
|
||||
- `os`: "iOS" or "Android"
|
||||
- `os_version`: "17.2.1", "14.0", etc.
|
||||
- `architecture`: "arm64", "aarch64"
|
||||
- `device_type`: "iPhone 15 Pro", "Samsung Galaxy S24", etc. (from device model identifier)
|
||||
|
||||
### Push Notifications
|
||||
|
||||
**iOS (APNs):**
|
||||
- App registers for push on launch: `UNUserNotificationCenter.requestAuthorization()`
|
||||
- Server stores APNs device token in `connect_machines.push_token`
|
||||
- When viewer requests session, server sends APNs push: `{"aps": {"alert": "Support session requested", "sound": "default"}, "session_id": "..."}`
|
||||
- User taps notification → app opens, auto-fills support code, prompts to start broadcast
|
||||
|
||||
**Android (FCM):**
|
||||
- App registers with Firebase on launch, uploads FCM token to server
|
||||
- Server sends FCM push when session requested
|
||||
- User taps notification → `MainActivity` opens with support code pre-filled
|
||||
|
||||
### Input Injection (Android Only)
|
||||
|
||||
**AccessibilityService setup:**
|
||||
1. User enables service in Settings → Accessibility → GuruConnect → toggle ON
|
||||
2. App declares service in `AndroidManifest.xml`:
|
||||
```xml
|
||||
<service android:name=".InputAccessibilityService"
|
||||
android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
|
||||
<intent-filter><action android:name="android.accessibilityservice.AccessibilityService" /></intent-filter>
|
||||
<meta-data android:name="android.accessibilityservice.config"
|
||||
android:resource="@xml/accessibility_service_config" />
|
||||
</service>
|
||||
```
|
||||
3. During session, relay sends `InputEvent` (touch/swipe) → service dispatches:
|
||||
```kotlin
|
||||
val path = Path().apply { moveTo(x, y) }
|
||||
val gesture = GestureDescription.Builder()
|
||||
.addStroke(GestureDescription.StrokeDescription(path, 0, duration))
|
||||
.build()
|
||||
dispatchGesture(gesture, null, null)
|
||||
```
|
||||
|
||||
**iOS: No input injection.** Relay server detects `mobile_capabilities.supports_input_injection = false` and disables input controls in viewer UI (show "View-Only Mode" banner).
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Files to Create
|
||||
|
||||
**iOS (Swift/SwiftUI):**
|
||||
- `GuruConnectMobile-iOS/App/GuruConnectApp.swift` — App entry, scene setup
|
||||
- `GuruConnectMobile-iOS/App/ContentView.swift` — Support code entry, connection status
|
||||
- `GuruConnectMobile-iOS/App/SessionView.swift` — Active session UI (timer, stop button)
|
||||
- `GuruConnectMobile-iOS/Broadcast/SampleHandler.swift` — ReplayKit extension, frame capture
|
||||
- `GuruConnectMobile-iOS/Broadcast/VideoEncoder.swift` — VideoToolbox H.264 encoding
|
||||
- `GuruConnectMobile-iOS/Shared/Transport/WebSocketClient.swift` — Starscream WebSocket wrapper
|
||||
- `GuruConnectMobile-iOS/Shared/Protocol/Protobuf.swift` — Swift protobuf codegen
|
||||
- `GuruConnectMobile-iOS/Shared/PushNotifications.swift` — APNs registration + handling
|
||||
|
||||
**Android (Kotlin/Jetpack Compose):**
|
||||
- `app/src/main/java/.../MainActivity.kt` — Compose UI, support code entry
|
||||
- `app/src/main/java/.../SessionActivity.kt` — Active session screen
|
||||
- `app/src/main/java/.../ScreenCaptureService.kt` — MediaProjection foreground service
|
||||
- `app/src/main/java/.../InputAccessibilityService.kt` — Accessibility service for input
|
||||
- `app/src/main/java/.../VideoEncoder.kt` — MediaCodec H.264 encoding
|
||||
- `app/src/main/java/.../WebSocketClient.kt` — OkHttp WebSocket
|
||||
- `app/src/main/java/.../ProtobufHandler.kt` — Protobuf serialization (protobuf-javalite)
|
||||
- `app/src/main/java/.../PushMessagingService.kt` — FCM message receiver
|
||||
|
||||
**Server (minor additions):**
|
||||
- `server/src/push/` — NEW module for APNs/FCM push sending
|
||||
- `server/src/push/apns.rs` — APNs HTTP/2 client (via `a2` crate)
|
||||
- `server/src/push/fcm.rs` — FCM HTTP v1 client
|
||||
- `server/src/push/mod.rs` — Unified `send_push_notification(device_token, session_id)` API
|
||||
- `server/src/api/devices.rs` — NEW: `POST /api/devices/:id/push-token` to store APNs/FCM tokens
|
||||
- `proto/guruconnect.proto` — Add `MobileCapabilities` and `TouchEvent` messages
|
||||
|
||||
**Shared:**
|
||||
- `proto/guruconnect.proto` — Update with mobile messages (protobuf source of truth)
|
||||
|
||||
### Key Logic
|
||||
|
||||
**iOS ReplayKit screen capture:**
|
||||
|
||||
```swift
|
||||
// GuruConnectMobile-iOS/Broadcast/SampleHandler.swift
|
||||
import ReplayKit
|
||||
import VideoToolbox
|
||||
|
||||
class SampleHandler: RPBroadcastSampleHandler {
|
||||
var encoder: VideoEncoder?
|
||||
var wsClient: WebSocketClient?
|
||||
|
||||
override func broadcastStarted(withSetupInfo setupInfo: [String : NSObject]?) {
|
||||
encoder = VideoEncoder()
|
||||
wsClient = WebSocketClient(url: "wss://connect.azcomputerguru.com/ws/agent")
|
||||
wsClient?.connect(supportCode: setupInfo?["supportCode"] as? String)
|
||||
}
|
||||
|
||||
override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
|
||||
guard sampleBufferType == .video else { return }
|
||||
|
||||
if let encoded = encoder?.encode(sampleBuffer) {
|
||||
let frameData = FrameData(data: encoded, width: 1920, height: 1080)
|
||||
wsClient?.send(frameData)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Android MediaProjection screen capture:**
|
||||
|
||||
```kotlin
|
||||
// ScreenCaptureService.kt
|
||||
class ScreenCaptureService : Service() {
|
||||
private lateinit var mediaProjection: MediaProjection
|
||||
private lateinit var virtualDisplay: VirtualDisplay
|
||||
private lateinit var imageReader: ImageReader
|
||||
private val encoder = VideoEncoder()
|
||||
private val wsClient = WebSocketClient()
|
||||
|
||||
override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
|
||||
val resultCode = intent?.getIntExtra("resultCode", 0) ?: return START_NOT_STICKY
|
||||
val data = intent.getParcelableExtra<Intent>("data") ?: return START_NOT_STICKY
|
||||
|
||||
val projection = MediaProjectionManager.getMediaProjection(resultCode, data)
|
||||
imageReader = ImageReader.newInstance(1920, 1080, PixelFormat.RGBA_8888, 2)
|
||||
|
||||
virtualDisplay = projection.createVirtualDisplay(
|
||||
"GuruConnect",
|
||||
1920, 1080, densityDpi,
|
||||
DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR,
|
||||
imageReader.surface, null, null
|
||||
)
|
||||
|
||||
imageReader.setOnImageAvailableListener({ reader ->
|
||||
val image = reader.acquireLatestImage()
|
||||
val encoded = encoder.encode(image)
|
||||
wsClient.sendFrame(encoded)
|
||||
image.close()
|
||||
}, backgroundHandler)
|
||||
|
||||
return START_STICKY
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Android Accessibility Service input injection:**
|
||||
|
||||
```kotlin
|
||||
// InputAccessibilityService.kt
|
||||
class InputAccessibilityService : AccessibilityService() {
|
||||
|
||||
fun injectTouch(x: Float, y: Float, action: TouchEvent.Action) {
|
||||
val displayMetrics = resources.displayMetrics
|
||||
val absX = x * displayMetrics.widthPixels
|
||||
val absY = y * displayMetrics.heightPixels
|
||||
|
||||
val path = Path().apply { moveTo(absX, absY) }
|
||||
val duration = if (action == TouchEvent.Action.DOWN || action == TouchEvent.Action.UP) 10L else 50L
|
||||
|
||||
val gesture = GestureDescription.Builder()
|
||||
.addStroke(GestureDescription.StrokeDescription(path, 0, duration))
|
||||
.build()
|
||||
|
||||
dispatchGesture(gesture, object : GestureResultCallback() {
|
||||
override fun onCompleted(gestureDescription: GestureDescription) {
|
||||
Log.d("GC", "Touch injected: ($absX, $absY)")
|
||||
}
|
||||
}, null)
|
||||
}
|
||||
|
||||
override fun onAccessibilityEvent(event: AccessibilityEvent?) {}
|
||||
override fun onInterrupt() {}
|
||||
}
|
||||
```
|
||||
|
||||
**Push notification handling (iOS):**
|
||||
|
||||
```swift
|
||||
// PushNotifications.swift
|
||||
import UserNotifications
|
||||
|
||||
class PushNotificationHandler: NSObject, UNUserNotificationCenterDelegate {
|
||||
func userNotificationCenter(_ center: UNUserNotificationCenter,
|
||||
didReceive response: UNNotificationResponse,
|
||||
withCompletionHandler completionHandler: @escaping () -> Void) {
|
||||
let userInfo = response.notification.request.content.userInfo
|
||||
if let sessionId = userInfo["session_id"] as? String {
|
||||
// Navigate to SessionView with pre-filled support code
|
||||
NotificationCenter.default.post(name: .sessionRequested, object: sessionId)
|
||||
}
|
||||
completionHandler()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### iOS Security
|
||||
|
||||
- **Screen Recording permission:** User must explicitly start ReplayKit broadcast; cannot be triggered remotely
|
||||
- **App Sandbox:** Extension runs in separate sandbox from main app; shared data via App Group only
|
||||
- **APNs authentication:** Server uses APNs auth key (`.p8` file) with Team ID + Key ID
|
||||
- **No input injection:** Not a security choice—iOS platform limitation (actually a security feature)
|
||||
|
||||
### Android Security
|
||||
|
||||
- **MediaProjection consent:** User must grant permission via system dialog; consent required per session (cannot be saved)
|
||||
- **Accessibility Service risk:** Granting Accessibility permission is high-privilege; app must clearly explain why (remote support) and warn user
|
||||
- **Foreground service:** Session runs as foreground service with persistent notification (user always aware)
|
||||
- **FCM authentication:** Server uses FCM service account key (JSON) for authenticated sends
|
||||
|
||||
### Authentication
|
||||
|
||||
**Support code only in v1:**
|
||||
- User enters 6-digit support code from dashboard
|
||||
- Agent authenticates via `POST /api/auth/support-code` (same as desktop agents)
|
||||
- Viewer token issued, session begins
|
||||
|
||||
**Persistent agent mode deferred to Phase 2:**
|
||||
- Requires secure storage of agent key (iOS Keychain, Android EncryptedSharedPreferences)
|
||||
- Requires background keep-alive (iOS: silent push, Android: foreground service)
|
||||
|
||||
### Privacy
|
||||
|
||||
- **Consent-first model:** User must actively grant screen sharing permission each session
|
||||
- **No background capture:** OS prevents capturing screen when app is backgrounded (security feature)
|
||||
- **User can stop anytime:** "Stop Sharing" button (iOS) or notification action (Android)
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
|
||||
- iOS: XCTest for protobuf serialization, support code validation
|
||||
- Android: JUnit + MockK for input event handling, encoder logic
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- **iOS test rig:** iPhone 14 Pro (iOS 17) physical device or simulator (ReplayKit requires real device for broadcast extension)
|
||||
- **Android test rig:** Pixel 6 (Android 14) physical device (MediaProjection requires real device)
|
||||
- Test flow: enter support code → grant permissions → viewer connects → verify frames received
|
||||
|
||||
### Manual Testing Scenarios
|
||||
|
||||
1. **iOS attended session:**
|
||||
- User opens app, enters support code, taps "Start Broadcast", selects GuruConnect
|
||||
- Viewer connects, sees iPhone home screen, user navigates Settings
|
||||
- Verify: frames display correctly, input controls disabled (view-only banner shown)
|
||||
- User taps "Stop Sharing" → session ends gracefully
|
||||
|
||||
2. **Android attended session with input:**
|
||||
- User opens app, enables Accessibility Service in Settings
|
||||
- User enters support code, grants MediaProjection permission
|
||||
- Viewer connects, remotely taps app icon, swipes, types text
|
||||
- Verify: input events execute on device, foreground notification shows
|
||||
- User swipes down notification, taps "Stop" → session ends
|
||||
|
||||
3. **Push notification wake:**
|
||||
- Viewer requests session from dashboard
|
||||
- Push notification appears on locked phone
|
||||
- User taps notification → app opens with support code pre-filled
|
||||
- User grants screen sharing → session starts
|
||||
|
||||
4. **Low bandwidth:** Throttle connection to 1 Mbps, verify H.264 adapts, frames remain usable
|
||||
|
||||
### App Store / Play Store Review
|
||||
|
||||
- **iOS App Store:** Requires detailed privacy policy explaining screen recording usage, ReplayKit justification in app review notes
|
||||
- **Google Play:** Requires Accessibility Service usage justification ("remote support for user's own device with their explicit consent")
|
||||
|
||||
## Effort Estimate & Dependencies
|
||||
|
||||
**Size:** X-Large (16-20 weeks, 1 developer with mobile experience)
|
||||
|
||||
**Breakdown:**
|
||||
- iOS app + ReplayKit extension: 5 weeks
|
||||
- Android app + MediaProjection service: 4 weeks
|
||||
- Android Accessibility Service input injection: 2 weeks
|
||||
- Push notification backend (APNs + FCM): 2 weeks
|
||||
- Server protobuf additions + mobile capabilities handling: 1 week
|
||||
- Viewer UI adjustments (touch event handling, view-only mode): 1 week
|
||||
- App Store + Play Store submission, review cycles: 2 weeks
|
||||
- Testing, edge cases, OS compatibility: 2 weeks
|
||||
- Buffer: 1-3 weeks
|
||||
|
||||
**Dependencies:**
|
||||
- **Apple Developer Program enrollment** ($99/year) — required for APNs + App Store distribution
|
||||
- **Google Play Developer account** ($25 one-time) — required for Play Store distribution
|
||||
- **Firebase project setup** (free tier) — for FCM push notifications
|
||||
- **SPEC-002 v2 Phase 1 completion** — per-agent keys model must be stable (already shipped)
|
||||
- **Mobile development expertise** — Swift/SwiftUI + Kotlin/Jetpack Compose; consider contract hire if not in-house
|
||||
|
||||
**Unblocks:**
|
||||
- Mobile support parity with competitors (ScreenConnect, TeamViewer, Splashtop all have mobile agents)
|
||||
- "Show me the problem" use case for phone/tablet support
|
||||
- BYOD enterprise support (employees request help on personal iOS/Android devices)
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **iOS view-only limitation — acceptable to market?** — Competitors (TeamViewer, Splashtop) also view-only on iOS due to platform constraints. Document prominently in UI/marketing.
|
||||
|
||||
2. **Android Accessibility Service friction — how to onboard?** — Most users don't know how to enable Accessibility. Need step-by-step wizard with screenshots. Alternatively: offer view-only Android mode (no Accessibility required) as Phase 1, add input in Phase 2.
|
||||
|
||||
3. **Foreground-only sessions — user can't multitask?** — Correct. iOS/Android stop screen capture when app backgrounds. This is an OS limitation, not a bug. Frame UI as "supervised support session" not "unattended monitoring."
|
||||
|
||||
4. **Push notification reliability?** — APNs/FCM are best-effort, not guaranteed. If push fails, user can manually open app and enter support code. Fallback: dashboard shows "waiting for device" with code to give user over phone.
|
||||
|
||||
5. **Cross-platform viewer compatibility?** — Existing native Windows viewer and web viewer already handle arbitrary frame sizes (mobile screens are just smaller). Touch events map to mouse clicks for non-touch-aware viewers.
|
||||
|
||||
6. **App Store/Play Store approval risk?** — Accessibility Service apps face extra scrutiny on Android. Emphasize "user-initiated remote support" positioning, not "remote monitoring." Provide detailed privacy policy. Low risk if framed correctly.
|
||||
|
||||
7. **Multi-touch support?** — v1 supports single-touch only (maps to mouse). Multi-touch (pinch-zoom, two-finger gestures) deferred to Phase 2 (requires `TouchEvent.pointer_id` array).
|
||||
|
||||
---
|
||||
|
||||
**Cross-references:**
|
||||
- GuruRMM SPEC-017: Mobile Device Support (MDM, inventory, lock/wipe) — complementary, not overlapping
|
||||
- SPEC-002: v2 modernization architecture (per-agent keys)
|
||||
- SPEC-010: Cross-platform agents (macOS/Linux) — similar platform abstraction approach
|
||||
- ADR-001: GuruConnect is standalone (no RMM coupling for this feature)
|
||||
931
docs/specs/SPEC-012-headless-linux-tty.md
Normal file
931
docs/specs/SPEC-012-headless-linux-tty.md
Normal file
@@ -0,0 +1,931 @@
|
||||
# SPEC-012: Headless Linux Mode (Serial Console + PTY Shell Access)
|
||||
|
||||
**Status:** Proposed
|
||||
**Priority:** P2
|
||||
**Requested By:** Mike Swanson (2026-05-30)
|
||||
**Estimated Effort:** Medium (5-7 weeks)
|
||||
|
||||
## Overview
|
||||
|
||||
Enable GuruConnect agent support for headless Linux servers (no X11/Wayland GUI) by providing two modes of terminal access: **Serial Console Mode** for boot-level access (GRUB, kernel messages, panics) and **PTY Shell Mode** for normal server management. This addresses critical server management use cases—from emergency recovery to routine administration—without requiring SSH. The viewer displays a terminal emulator (xterm.js web viewer) connected to either the system serial console (`/dev/ttyS0`) or a pseudo-TTY shell session. Serial Console Mode provides true "remote console" access like KVM-over-IP or IPMI Serial-over-LAN, seeing everything the physical monitor would show. PTY Shell Mode provides an interactive shell for normal management tasks. Success criteria: technician can access GRUB bootloader, view kernel boot messages, handle kernel panics, AND perform routine server management—all via GuruConnect dashboard with centralized authentication and audit logging.
|
||||
|
||||
**Use Cases:**
|
||||
- **Boot-level access:** GRUB menu selection, kernel parameter editing, single-user mode
|
||||
- **Emergency recovery:** Kernel panic diagnosis, filesystem repair, systemd rescue shell
|
||||
- **Server management:** Package updates, configuration changes, log review (normal shell access)
|
||||
- **Container debugging:** Exec into running containers via GuruConnect
|
||||
- **MSP consolidation:** One tool for desktop support (GUI), server boot recovery (console), and server management (shell)
|
||||
|
||||
**Success Criteria:**
|
||||
- **Serial Console Mode:** View GRUB bootloader, kernel boot messages, kernel panics, login prompts—as if sitting at physical console
|
||||
- **PTY Shell Mode:** Interactive shell (bash/zsh) with full ANSI color, cursor control, vim/nano/htop support
|
||||
- GuruConnect agent runs on Ubuntu Server 22.04 minimal install (no desktop packages)
|
||||
- Dashboard mode selector: "Console" vs. "Shell" per agent (user chooses at connection time)
|
||||
- Same protobuf-over-WSS transport, support-code and persistent-agent authentication
|
||||
- Audit logging: session recording for both console and shell modes
|
||||
|
||||
## Scope
|
||||
|
||||
### Included in v1
|
||||
|
||||
**Mode 1: Serial Console Mode (True Remote Console)**
|
||||
- Open system serial console device (`/dev/ttyS0` or `/dev/console`) for raw I/O
|
||||
- Relay all bytes bidirectionally: console output → `TerminalData` → viewer; viewer input → `TerminalInput` → console
|
||||
- **Sees everything:** GRUB bootloader menu, kernel boot messages, systemd startup, login prompts, kernel panics
|
||||
- **Boot-time interaction:** Select GRUB entries, edit kernel parameters, boot into single-user mode
|
||||
- Requires root privileges (serial console access restricted to root)
|
||||
- Requires serial console enabled on target server (GRUB + kernel parameters configured)
|
||||
- No PTY spawning—direct device I/O, like `screen /dev/ttyS0 115200`
|
||||
- Agent config flag: `console_mode: true` + `console_device: "/dev/ttyS0"`
|
||||
|
||||
**Mode 2: PTY Shell Mode (Interactive Shell)**
|
||||
- Detect headless environment (no DISPLAY, no X11/Wayland libraries) at runtime
|
||||
- Spawn pseudo-TTY (PTY) via `openpty()` + fork/exec shell (`/bin/bash -l` or user's `$SHELL`)
|
||||
- Terminal I/O: read PTY output → encode as protobuf `TerminalData` → send via WebSocket
|
||||
- Input: receive protobuf `TerminalInput` → write to PTY master
|
||||
- Terminal resize: handle `TerminalResize` message → send `SIGWINCH` to PTY
|
||||
- Fallback shell selection: `$SHELL` env var → `/bin/bash` → `/bin/sh`
|
||||
- Graceful PTY cleanup on session end (send exit command, wait for shell exit, close PTY)
|
||||
- Standard user privileges (runs as agent service user)
|
||||
|
||||
**Mode Selection:**
|
||||
- Dashboard shows mode selector when connecting to headless agent: "Console" vs. "Shell"
|
||||
- "Console" button: viewer sends `mode: console` in connection request → agent opens `/dev/ttyS0`
|
||||
- "Shell" button: viewer sends `mode: shell` in connection request → agent spawns PTY
|
||||
- Agent config specifies default mode if serial console unavailable
|
||||
- If serial console device doesn't exist or permission denied, fall back to PTY shell mode with warning
|
||||
|
||||
**Both Modes Share:**
|
||||
- Same agent binary: `guruconnect` detects headless and offers both modes
|
||||
- Same xterm.js viewer (handles both serial console and PTY identically)
|
||||
|
||||
**Viewer (Web Viewer):**
|
||||
- xterm.js-based terminal emulator embedded in `viewer.html`
|
||||
- Connects to same `/ws/viewer` endpoint with session JWT
|
||||
- Relay server detects `TerminalData` frames (not `FrameData`) and routes accordingly
|
||||
- Terminal controls: resize on window resize, copy/paste support, configurable font size
|
||||
- Session toolbar: connection status, terminal size (e.g., "80x24"), reconnect button
|
||||
|
||||
**Viewer (Native Desktop Viewer - optional Phase 2):**
|
||||
- Defer native viewer terminal support to Phase 2
|
||||
- v1: web viewer only for terminal sessions (show "Open in browser" prompt if launched via `guruconnect://`)
|
||||
|
||||
**Protobuf Protocol:**
|
||||
- New message types: `TerminalData` (PTY output), `TerminalInput` (keyboard input), `TerminalResize` (window size)
|
||||
- `AgentStatus` includes `terminal_mode: bool` flag (true for headless agents)
|
||||
- Dashboard shows terminal icon for headless agents, camera icon for GUI agents
|
||||
|
||||
**Dashboard:**
|
||||
- Detect `terminal_mode: true` in agent status
|
||||
- "Connect" button opens web viewer in terminal mode (not screen capture mode)
|
||||
- Agent list shows "Terminal" badge for headless agents
|
||||
|
||||
**Session Recording (Audit):**
|
||||
- Log all terminal I/O to `events` table or separate `terminal_sessions` table
|
||||
- Playback: recorded session can be replayed as "terminal recording" (asciicast format or raw PTY dump)
|
||||
|
||||
### Explicitly out of scope
|
||||
|
||||
- **GUI mode on headless agents** — v1 is terminal-only; no attempt to start Xvfb or launch GUI apps
|
||||
- **SSH key management** — agent uses GuruConnect auth (support code / agent key), not SSH keys
|
||||
- **File transfer via terminal** — defer to SPEC (file transfer is a separate roadmap item for all agent types)
|
||||
- **Multi-user terminal sessions** — v1 is single-session console/PTY; no tmux/screen built-in sharing
|
||||
- **Windows terminal mode** — defer; Windows Server typically has GUI (RDP) or SSH (OpenSSH)
|
||||
- **macOS terminal mode** — defer; macOS servers are rare and typically have GUI access
|
||||
- **Framebuffer capture (`/dev/fb0`)** — defer; serial console is more reliable and doesn't require framebuffer device
|
||||
|
||||
### Serial Console Setup Requirements (Mode 1)
|
||||
|
||||
To use Serial Console Mode, the target Linux server must be configured to output to serial console. This is a **one-time setup per server** (typically done during provisioning):
|
||||
|
||||
**Step 1: Configure GRUB**
|
||||
|
||||
Edit `/etc/default/grub`:
|
||||
```bash
|
||||
# Enable serial console output at 115200 baud
|
||||
GRUB_TERMINAL="serial console"
|
||||
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"
|
||||
|
||||
# Kernel console output to both VGA (tty0) and serial (ttyS0)
|
||||
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"
|
||||
```
|
||||
|
||||
Update GRUB and reboot:
|
||||
```bash
|
||||
sudo update-grub # Debian/Ubuntu
|
||||
# OR
|
||||
sudo grub2-mkconfig -o /boot/grub2/grub.cfg # RHEL/CentOS
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
**Step 2: Enable getty on Serial Console**
|
||||
|
||||
Ensure a login prompt appears on serial console after boot:
|
||||
```bash
|
||||
sudo systemctl enable serial-getty@ttyS0.service
|
||||
sudo systemctl start serial-getty@ttyS0.service
|
||||
```
|
||||
|
||||
**Step 3: Verify**
|
||||
|
||||
Test serial console locally before configuring GuruConnect:
|
||||
```bash
|
||||
sudo screen /dev/ttyS0 115200
|
||||
# Should see kernel messages, login prompt
|
||||
```
|
||||
|
||||
**What This Provides:**
|
||||
- ✓ GRUB bootloader menu visible via serial console
|
||||
- ✓ Kernel boot messages stream to serial console
|
||||
- ✓ Login prompt on `/dev/ttyS0` after boot
|
||||
- ✓ Kernel panics output to serial console
|
||||
- ✓ systemd rescue shell accessible via serial console
|
||||
|
||||
**Compatibility:**
|
||||
- Physical servers: Uses hardware serial port (COM1 = ttyS0)
|
||||
- Virtual machines: VMware/Proxmox/KVM expose virtual serial port; configure VM to attach serial port
|
||||
- Cloud VMs: AWS, GCP, Azure offer "Serial Console" feature (already configured); GuruConnect agent can relay it
|
||||
|
||||
## Architecture
|
||||
|
||||
### Agent Mode Selection
|
||||
|
||||
**Connection request handling:**
|
||||
|
||||
```rust
|
||||
// agent/src/session/terminal.rs
|
||||
pub async fn handle_terminal_session(
|
||||
ws: WebSocketClient,
|
||||
mode: TerminalMode, // Console or Shell
|
||||
support_code: String
|
||||
) -> Result<()> {
|
||||
match mode {
|
||||
TerminalMode::Console => run_console_session(ws, support_code).await,
|
||||
TerminalMode::Shell => run_shell_session(ws, support_code).await,
|
||||
}
|
||||
}
|
||||
|
||||
pub enum TerminalMode {
|
||||
Console, // Serial console (/dev/ttyS0)
|
||||
Shell, // PTY shell session
|
||||
}
|
||||
```
|
||||
|
||||
### Agent Serial Console Handling (Mode 1)
|
||||
|
||||
**Serial device open:**
|
||||
|
||||
```rust
|
||||
// agent/src/platform/linux/console.rs
|
||||
use std::fs::OpenOptions;
|
||||
use std::os::unix::io::AsRawFd;
|
||||
|
||||
pub struct ConsoleSession {
|
||||
device_fd: RawFd,
|
||||
device_path: String, // "/dev/ttyS0" or "/dev/console"
|
||||
}
|
||||
|
||||
impl ConsoleSession {
|
||||
pub fn open(device_path: &str) -> Result<Self> {
|
||||
// Open serial console device for read/write
|
||||
// Requires root privileges
|
||||
let file = OpenOptions::new()
|
||||
.read(true)
|
||||
.write(true)
|
||||
.open(device_path)
|
||||
.context("Failed to open serial console - requires root")?;
|
||||
|
||||
let device_fd = file.as_raw_fd();
|
||||
|
||||
// Configure terminal settings (115200 baud, 8N1)
|
||||
unsafe {
|
||||
let mut termios: libc::termios = std::mem::zeroed();
|
||||
if libc::tcgetattr(device_fd, &mut termios) != 0 {
|
||||
return Err(anyhow!("tcgetattr failed"));
|
||||
}
|
||||
|
||||
// Set baud rate to 115200
|
||||
libc::cfsetispeed(&mut termios, libc::B115200);
|
||||
libc::cfsetospeed(&mut termios, libc::B115200);
|
||||
|
||||
// 8N1 (8 data bits, no parity, 1 stop bit)
|
||||
termios.c_cflag &= !libc::CSIZE;
|
||||
termios.c_cflag |= libc::CS8;
|
||||
termios.c_cflag &= !(libc::PARENB | libc::PARODD);
|
||||
termios.c_cflag &= !libc::CSTOPB;
|
||||
|
||||
// Raw mode (no line buffering, no echo)
|
||||
libc::cfmakeraw(&mut termios);
|
||||
|
||||
if libc::tcsetattr(device_fd, libc::TCSANOW, &termios) != 0 {
|
||||
return Err(anyhow!("tcsetattr failed"));
|
||||
}
|
||||
}
|
||||
|
||||
Ok(ConsoleSession {
|
||||
device_fd,
|
||||
device_path: device_path.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn read(&self, buf: &mut [u8]) -> Result<usize> {
|
||||
unsafe {
|
||||
let n = libc::read(self.device_fd, buf.as_mut_ptr() as *mut _, buf.len());
|
||||
if n < 0 {
|
||||
Err(anyhow!("Console read failed"))
|
||||
} else {
|
||||
Ok(n as usize)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn write(&self, data: &[u8]) -> Result<()> {
|
||||
unsafe {
|
||||
let n = libc::write(self.device_fd, data.as_ptr() as *const _, data.len());
|
||||
if n < 0 {
|
||||
Err(anyhow!("Console write failed"))
|
||||
} else {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for ConsoleSession {
|
||||
fn drop(&mut self) {
|
||||
unsafe { libc::close(self.device_fd); }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Console session loop:**
|
||||
|
||||
```rust
|
||||
// agent/src/session/console.rs
|
||||
pub async fn run_console_session(ws: WebSocketClient, support_code: String) -> Result<()> {
|
||||
// Try /dev/ttyS0 first, fall back to /dev/console
|
||||
let console = ConsoleSession::open("/dev/ttyS0")
|
||||
.or_else(|_| ConsoleSession::open("/dev/console"))?;
|
||||
|
||||
// Status update: terminal mode, console
|
||||
ws.send(AgentStatus {
|
||||
terminal_mode: true,
|
||||
console_mode: true, // NEW flag
|
||||
os: "Linux".to_string(),
|
||||
// ...
|
||||
}).await?;
|
||||
|
||||
let mut buf = vec![0u8; 4096];
|
||||
loop {
|
||||
tokio::select! {
|
||||
// Read console output, send to relay
|
||||
Ok(n) = tokio::task::spawn_blocking({
|
||||
let fd = console.device_fd;
|
||||
move || unsafe { libc::read(fd, buf.as_mut_ptr() as *mut _, buf.len()) }
|
||||
}) => {
|
||||
if n > 0 {
|
||||
ws.send(TerminalData {
|
||||
data: buf[..n as usize].to_vec(),
|
||||
}).await?;
|
||||
}
|
||||
}
|
||||
|
||||
// Receive input from relay, write to console
|
||||
Some(msg) = ws.recv() => {
|
||||
match msg {
|
||||
Message::TerminalInput(input) => {
|
||||
console.write(&input.data)?;
|
||||
}
|
||||
Message::Disconnect => break,
|
||||
// Note: Resize ignored for serial console (not applicable)
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
### Agent PTY Handling (Mode 2)
|
||||
|
||||
**Headless detection:**
|
||||
|
||||
```rust
|
||||
// agent/src/platform/linux/headless.rs
|
||||
pub fn is_headless() -> bool {
|
||||
// Check if DISPLAY is unset and no X11/Wayland session detected
|
||||
std::env::var("DISPLAY").is_err() &&
|
||||
std::env::var("WAYLAND_DISPLAY").is_err() &&
|
||||
!std::path::Path::new("/tmp/.X11-unix").exists()
|
||||
}
|
||||
```
|
||||
|
||||
**PTY spawn:**
|
||||
|
||||
```rust
|
||||
// agent/src/platform/linux/pty.rs
|
||||
use libc::{openpty, fork, execvp, dup2, STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO, winsize, TIOCSWINSZ};
|
||||
use std::os::unix::io::RawFd;
|
||||
|
||||
pub struct PtySession {
|
||||
master_fd: RawFd,
|
||||
child_pid: libc::pid_t,
|
||||
cols: u16,
|
||||
rows: u16,
|
||||
}
|
||||
|
||||
impl PtySession {
|
||||
pub fn spawn(shell: &str, cols: u16, rows: u16) -> Result<Self> {
|
||||
let mut master_fd: RawFd = 0;
|
||||
let mut slave_fd: RawFd = 0;
|
||||
let mut winsize = winsize {
|
||||
ws_row: rows,
|
||||
ws_col: cols,
|
||||
ws_xpixel: 0,
|
||||
ws_ypixel: 0,
|
||||
};
|
||||
|
||||
unsafe {
|
||||
if openpty(&mut master_fd, &mut slave_fd, std::ptr::null_mut(),
|
||||
std::ptr::null(), &mut winsize as *mut _) != 0 {
|
||||
return Err(anyhow!("openpty failed"));
|
||||
}
|
||||
|
||||
let pid = fork();
|
||||
if pid == 0 {
|
||||
// Child process: exec shell
|
||||
dup2(slave_fd, STDIN_FILENO);
|
||||
dup2(slave_fd, STDOUT_FILENO);
|
||||
dup2(slave_fd, STDERR_FILENO);
|
||||
libc::close(master_fd);
|
||||
libc::close(slave_fd);
|
||||
|
||||
let shell_cstr = CString::new(shell)?;
|
||||
let args = [shell_cstr.as_ptr(), std::ptr::null()];
|
||||
execvp(shell_cstr.as_ptr(), args.as_ptr());
|
||||
std::process::exit(1); // exec failed
|
||||
} else {
|
||||
// Parent process: close slave, return master FD
|
||||
libc::close(slave_fd);
|
||||
Ok(PtySession {
|
||||
master_fd,
|
||||
child_pid: pid,
|
||||
cols,
|
||||
rows,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn read(&self, buf: &mut [u8]) -> Result<usize> {
|
||||
unsafe {
|
||||
let n = libc::read(self.master_fd, buf.as_mut_ptr() as *mut _, buf.len());
|
||||
if n < 0 {
|
||||
Err(anyhow!("PTY read failed"))
|
||||
} else {
|
||||
Ok(n as usize)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn write(&self, data: &[u8]) -> Result<()> {
|
||||
unsafe {
|
||||
let n = libc::write(self.master_fd, data.as_ptr() as *const _, data.len());
|
||||
if n < 0 {
|
||||
Err(anyhow!("PTY write failed"))
|
||||
} else {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn resize(&mut self, cols: u16, rows: u16) -> Result<()> {
|
||||
self.cols = cols;
|
||||
self.rows = rows;
|
||||
let winsize = winsize {
|
||||
ws_row: rows,
|
||||
ws_col: cols,
|
||||
ws_xpixel: 0,
|
||||
ws_ypixel: 0,
|
||||
};
|
||||
unsafe {
|
||||
if libc::ioctl(self.master_fd, TIOCSWINSZ, &winsize as *const _) != 0 {
|
||||
return Err(anyhow!("TIOCSWINSZ failed"));
|
||||
}
|
||||
}
|
||||
// Send SIGWINCH to child process group
|
||||
unsafe { libc::kill(-self.child_pid, libc::SIGWINCH); }
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for PtySession {
|
||||
fn drop(&mut self) {
|
||||
unsafe {
|
||||
libc::close(self.master_fd);
|
||||
// Send SIGTERM to child, wait briefly, then SIGKILL if still alive
|
||||
libc::kill(self.child_pid, libc::SIGTERM);
|
||||
std::thread::sleep(std::time::Duration::from_millis(500));
|
||||
libc::waitpid(self.child_pid, std::ptr::null_mut(), libc::WNOHANG);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Agent session loop:**
|
||||
|
||||
```rust
|
||||
// agent/src/session/terminal.rs
|
||||
pub async fn run_terminal_session(ws: WebSocketClient, support_code: String) -> Result<()> {
|
||||
let shell = std::env::var("SHELL").unwrap_or_else(|_| "/bin/bash".to_string());
|
||||
let mut pty = PtySession::spawn(&shell, 80, 24)?;
|
||||
|
||||
// Status update: terminal mode
|
||||
ws.send(AgentStatus {
|
||||
terminal_mode: true,
|
||||
os: "Linux".to_string(),
|
||||
// ...
|
||||
}).await?;
|
||||
|
||||
let mut buf = vec![0u8; 4096];
|
||||
loop {
|
||||
tokio::select! {
|
||||
// Read PTY output, send to relay
|
||||
Ok(n) = tokio::task::spawn_blocking({
|
||||
let master = pty.master_fd;
|
||||
move || unsafe { libc::read(master, buf.as_mut_ptr() as *mut _, buf.len()) }
|
||||
}) => {
|
||||
if n > 0 {
|
||||
ws.send(TerminalData {
|
||||
data: buf[..n as usize].to_vec(),
|
||||
}).await?;
|
||||
}
|
||||
}
|
||||
|
||||
// Receive input from relay, write to PTY
|
||||
Some(msg) = ws.recv() => {
|
||||
match msg {
|
||||
Message::TerminalInput(input) => {
|
||||
pty.write(&input.data)?;
|
||||
}
|
||||
Message::TerminalResize(resize) => {
|
||||
pty.resize(resize.cols, resize.rows)?;
|
||||
}
|
||||
Message::Disconnect => break,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
### Protobuf Protocol Extensions
|
||||
|
||||
```protobuf
|
||||
// proto/guruconnect.proto
|
||||
|
||||
message AgentStatus {
|
||||
// Existing fields...
|
||||
optional bool terminal_mode = 21; // true for headless agents
|
||||
optional bool console_mode = 22; // true for serial console mode, false for PTY shell mode
|
||||
}
|
||||
|
||||
message TerminalData {
|
||||
bytes data = 1; // Raw terminal output (PTY or serial console, includes ANSI escape sequences)
|
||||
}
|
||||
|
||||
message TerminalInput {
|
||||
bytes data = 1; // Keyboard input from viewer (UTF-8 encoded)
|
||||
}
|
||||
|
||||
message TerminalResize {
|
||||
uint32 cols = 1; // Terminal width (characters)
|
||||
uint32 rows = 2; // Terminal height (lines)
|
||||
// Note: Resize only applies to PTY shell mode; serial console ignores this
|
||||
}
|
||||
|
||||
enum TerminalModeRequest {
|
||||
SHELL = 0; // Request PTY shell session
|
||||
CONSOLE = 1; // Request serial console session
|
||||
}
|
||||
|
||||
message SessionRequest {
|
||||
// Existing fields...
|
||||
optional TerminalModeRequest terminal_mode_request = 10; // NEW: viewer specifies console vs. shell
|
||||
}
|
||||
|
||||
// Update AgentMessage and ViewerMessage unions
|
||||
message AgentMessage {
|
||||
oneof message {
|
||||
AgentStatus status = 1;
|
||||
FrameData frame = 2;
|
||||
TerminalData terminal_data = 10; // NEW
|
||||
}
|
||||
}
|
||||
|
||||
message ViewerMessage {
|
||||
oneof message {
|
||||
InputEvent input = 1;
|
||||
TerminalInput terminal_input = 10; // NEW
|
||||
TerminalResize terminal_resize = 11; // NEW
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Relay Server Changes
|
||||
|
||||
**Route terminal vs. screen capture sessions:**
|
||||
|
||||
```rust
|
||||
// server/src/relay/mod.rs
|
||||
async fn handle_agent_message(msg: AgentMessage, session: &Session) {
|
||||
match msg.message {
|
||||
Some(agent_message::Message::Status(status)) => {
|
||||
session.terminal_mode = status.terminal_mode.unwrap_or(false);
|
||||
// Store in DB: UPDATE sessions SET terminal_mode = ? WHERE id = ?
|
||||
}
|
||||
Some(agent_message::Message::TerminalData(data)) => {
|
||||
// Forward to viewer WebSocket
|
||||
if let Some(viewer_ws) = session.viewer_ws.lock().await.as_mut() {
|
||||
viewer_ws.send(ViewerMessage {
|
||||
message: Some(viewer_message::Message::TerminalData(data))
|
||||
}).await?;
|
||||
}
|
||||
// Optional: append to terminal_recording buffer for audit
|
||||
}
|
||||
Some(agent_message::Message::Frame(frame)) => {
|
||||
// Existing screen capture logic...
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
|
||||
async fn handle_viewer_message(msg: ViewerMessage, session: &Session) {
|
||||
match msg.message {
|
||||
Some(viewer_message::Message::TerminalInput(input)) => {
|
||||
// Forward to agent WebSocket
|
||||
if let Some(agent_ws) = session.agent_ws.lock().await.as_mut() {
|
||||
agent_ws.send(AgentMessage {
|
||||
message: Some(agent_message::Message::TerminalInput(input))
|
||||
}).await?;
|
||||
}
|
||||
}
|
||||
Some(viewer_message::Message::TerminalResize(resize)) => {
|
||||
// Forward resize to agent
|
||||
if let Some(agent_ws) = session.agent_ws.lock().await.as_mut() {
|
||||
agent_ws.send(AgentMessage {
|
||||
message: Some(agent_message::Message::TerminalResize(resize))
|
||||
}).await?;
|
||||
}
|
||||
}
|
||||
Some(viewer_message::Message::Input(input)) => {
|
||||
// Existing GUI input logic...
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Web Viewer (xterm.js)
|
||||
|
||||
**HTML template:**
|
||||
|
||||
```html
|
||||
<!-- server/static/viewer-terminal.html -->
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<title>GuruConnect Terminal</title>
|
||||
<link rel="stylesheet" href="/vendor/xterm/xterm.css" />
|
||||
<script src="/vendor/xterm/xterm.js"></script>
|
||||
<script src="/vendor/xterm/xterm-addon-fit.js"></script>
|
||||
<style>
|
||||
#terminal { height: 100vh; }
|
||||
.toolbar { background: #333; color: #fff; padding: 8px; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="toolbar">
|
||||
<span id="status">Connecting...</span>
|
||||
<span id="size" style="float: right;">80x24</span>
|
||||
</div>
|
||||
<div id="terminal"></div>
|
||||
<script>
|
||||
const term = new Terminal({
|
||||
cursorBlink: true,
|
||||
fontSize: 14,
|
||||
fontFamily: 'Consolas, "Courier New", monospace',
|
||||
theme: {
|
||||
background: '#1e1e1e',
|
||||
foreground: '#d4d4d4',
|
||||
}
|
||||
});
|
||||
const fitAddon = new FitAddon.FitAddon();
|
||||
term.loadAddon(fitAddon);
|
||||
term.open(document.getElementById('terminal'));
|
||||
fitAddon.fit();
|
||||
|
||||
const ws = new WebSocket(`wss://${location.host}/ws/viewer?token=${TOKEN}&session=${SESSION_ID}`);
|
||||
|
||||
ws.onopen = () => {
|
||||
document.getElementById('status').textContent = 'Connected';
|
||||
// Send initial terminal size
|
||||
ws.send(encodeTerminalResize(term.cols, term.rows));
|
||||
};
|
||||
|
||||
ws.onmessage = (event) => {
|
||||
const msg = decodeProtobuf(event.data);
|
||||
if (msg.terminal_data) {
|
||||
term.write(new Uint8Array(msg.terminal_data.data));
|
||||
}
|
||||
};
|
||||
|
||||
term.onData((data) => {
|
||||
ws.send(encodeTerminalInput(data));
|
||||
});
|
||||
|
||||
term.onResize(({ cols, rows }) => {
|
||||
document.getElementById('size').textContent = `${cols}x${rows}`;
|
||||
ws.send(encodeTerminalResize(cols, rows));
|
||||
});
|
||||
|
||||
window.addEventListener('resize', () => fitAddon.fit());
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
```
|
||||
|
||||
### Dashboard Mode Selector
|
||||
|
||||
```javascript
|
||||
// server/static/dashboard.js
|
||||
function renderAgentRow(agent) {
|
||||
const icon = agent.terminal_mode
|
||||
? '<i class="icon-terminal"></i> Terminal'
|
||||
: '<i class="icon-screen"></i> Screen';
|
||||
|
||||
// For headless agents, show mode selector (Console vs. Shell)
|
||||
let connectButtons;
|
||||
if (agent.terminal_mode && agent.online) {
|
||||
connectButtons = `
|
||||
<div class="terminal-mode-selector">
|
||||
<button class="btn-console" onclick="connectToTerminal('${agent.id}', 'console')"
|
||||
title="Serial console access (GRUB, boot, panics)">
|
||||
Console
|
||||
</button>
|
||||
<button class="btn-shell" onclick="connectToTerminal('${agent.id}', 'shell')"
|
||||
title="Interactive shell (bash/zsh)">
|
||||
Shell
|
||||
</button>
|
||||
</div>
|
||||
`;
|
||||
} else if (!agent.terminal_mode && agent.online) {
|
||||
// GUI agent
|
||||
connectButtons = `<button onclick="connectToAgent('${agent.id}')">Connect</button>`;
|
||||
} else {
|
||||
connectButtons = '<span>Offline</span>';
|
||||
}
|
||||
|
||||
return `<tr>
|
||||
<td>${agent.name}</td>
|
||||
<td>${icon}</td>
|
||||
<td>${agent.os} ${agent.os_version}</td>
|
||||
<td>${connectButtons}</td>
|
||||
</tr>`;
|
||||
}
|
||||
|
||||
function connectToAgent(agentId) {
|
||||
// GUI agent connection
|
||||
window.open(`/viewer.html?session=${agentId}&token=${JWT}`, '_blank');
|
||||
}
|
||||
|
||||
function connectToTerminal(agentId, mode) {
|
||||
// Terminal agent connection with mode parameter
|
||||
window.open(`/viewer-terminal.html?session=${agentId}&token=${JWT}&mode=${mode}`, '_blank');
|
||||
}
|
||||
```
|
||||
|
||||
**Dashboard UI for headless agents:**
|
||||
- Shows two buttons: "Console" and "Shell"
|
||||
- "Console" button: opens serial console session (GRUB, boot messages, panics)
|
||||
- "Shell" button: opens PTY shell session (normal server management)
|
||||
- Tooltip on hover explains each mode
|
||||
- Mode parameter passed to viewer via URL query string
|
||||
|
||||
### Database Schema
|
||||
|
||||
**Minor addition to `sessions` table:**
|
||||
|
||||
```sql
|
||||
-- migrations/012_terminal_mode.sql
|
||||
ALTER TABLE connect_sessions ADD COLUMN terminal_mode BOOLEAN DEFAULT FALSE;
|
||||
|
||||
-- Optional: separate table for terminal recordings
|
||||
CREATE TABLE IF NOT EXISTS terminal_recordings (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
session_id UUID REFERENCES connect_sessions(id) ON DELETE CASCADE,
|
||||
started_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
ended_at TIMESTAMPTZ,
|
||||
recording_data BYTEA, -- asciicast JSON or raw PTY dump (compressed)
|
||||
size_bytes BIGINT,
|
||||
INDEX idx_terminal_recordings_session (session_id)
|
||||
);
|
||||
```
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Files to Create
|
||||
|
||||
**Agent (Linux-specific):**
|
||||
- `agent/src/platform/linux/console.rs` — NEW: Serial console device I/O (`/dev/ttyS0`, termios config)
|
||||
- `agent/src/session/console.rs` — NEW: Console session loop (serial device ↔ WebSocket)
|
||||
- `agent/src/platform/linux/pty.rs` — PTY spawn, I/O, resize (openpty, fork, exec)
|
||||
- `agent/src/platform/linux/headless.rs` — Headless detection logic
|
||||
- `agent/src/session/terminal.rs` — Mode dispatcher (console vs. shell), shell session loop
|
||||
|
||||
**Server:**
|
||||
- `server/src/relay/terminal.rs` — Terminal message routing (TerminalData/Input/Resize)
|
||||
- `server/static/viewer-terminal.html` — xterm.js-based web terminal viewer
|
||||
- `server/static/vendor/xterm/` — xterm.js library files (CDN or bundled)
|
||||
- `server/migrations/012_terminal_mode.sql` — Schema update
|
||||
|
||||
**Protobuf:**
|
||||
- `proto/guruconnect.proto` — Add TerminalData, TerminalInput, TerminalResize messages
|
||||
|
||||
**Dashboard:**
|
||||
- `server/static/dashboard.js` — Detect `terminal_mode`, render terminal icon, route to terminal viewer
|
||||
|
||||
### Key Dependencies
|
||||
|
||||
```toml
|
||||
# agent/Cargo.toml (Linux-specific)
|
||||
[target.'cfg(target_os = "linux")'.dependencies]
|
||||
libc = "0.2" # openpty, fork, exec, ioctl
|
||||
nix = "0.27" # Safe wrappers for POSIX APIs
|
||||
```
|
||||
|
||||
**xterm.js (web viewer):**
|
||||
- Version: 5.3.0+ (latest stable)
|
||||
- Addons: `xterm-addon-fit` (auto-resize)
|
||||
- Delivery: CDN link or bundled in `server/static/vendor/xterm/`
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Serial Console Access (Mode 1)
|
||||
|
||||
- **Requires root privileges:** Opening `/dev/ttyS0` or `/dev/console` requires root access
|
||||
- **Implication:** Agent must run as root for console mode, OR use capabilities (`CAP_SYS_TTY_CONFIG`)
|
||||
- **Boot-level control:** Serial console grants full boot-time control (GRUB menu, kernel parameters, single-user mode)
|
||||
- **Risk:** Attacker with console access can modify bootloader, disable security features, boot into recovery
|
||||
- **Mitigation 1:** Restrict console mode to authorized users only (dashboard RBAC: "console_access" permission)
|
||||
- **Mitigation 2:** Require MFA for console mode sessions (stronger auth than shell mode)
|
||||
- **Mitigation 3:** Audit logging: record ALL console I/O with immutable timestamps
|
||||
- **Mitigation 4:** Alert on console mode connections (notify admin when console session starts)
|
||||
|
||||
**Recommended deployment:**
|
||||
- Run agent as unprivileged user for shell mode (default)
|
||||
- For console mode: either run agent as root OR grant `CAP_SYS_TTY_CONFIG` capability via systemd unit
|
||||
|
||||
### Shell Access Risk (Mode 2)
|
||||
|
||||
- **Privilege escalation:** PTY spawns shell as the agent's user (typically unprivileged `guruconnect` service user)
|
||||
- **Mitigation 1:** Run agent as unprivileged user, use `sudo` for privileged commands
|
||||
- **Mitigation 2:** Add `allowed_commands` whitelist (optional Phase 2 feature) — restrict to specific binaries
|
||||
- **Mitigation 3:** Audit logging: record all terminal I/O for compliance review
|
||||
|
||||
### Authentication
|
||||
|
||||
**Same as GUI agents:**
|
||||
- Support-code for ad-hoc sessions (6-digit, time-limited)
|
||||
- Persistent agent key for managed servers (per-agent `cak_*` key from SPEC-004)
|
||||
- Viewer JWT token required for WebSocket connection
|
||||
|
||||
### Session Recording (Compliance)
|
||||
|
||||
- **Optional toggle:** dashboard setting "Record terminal sessions" (default: ON for compliance)
|
||||
- **Storage:** `terminal_recordings` table (BYTEA column, compressed)
|
||||
- **Playback:** Admin dashboard can replay terminal sessions as asciicast (xterm.js built-in playback)
|
||||
- **Retention:** configurable (default: 90 days, auto-purge older recordings)
|
||||
|
||||
### Input Sanitization
|
||||
|
||||
- **No sanitization needed:** PTY handles raw bytes; ANSI escape sequences are terminal-native
|
||||
- **DoS risk:** Malicious viewer could spam resize events; rate-limit `TerminalResize` (max 10/sec)
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
|
||||
- PTY spawn/cleanup: verify `openpty()` success, shell exec, FD management
|
||||
- Terminal I/O: mock PTY master FD, test read/write buffers
|
||||
- Protobuf serialization: TerminalData/Input/Resize round-trip
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- **Headless VM:** Ubuntu Server 22.04 minimal (no desktop packages)
|
||||
- **Agent install:** `guruconnect` binary, systemd service, no X11 deps
|
||||
- **Connect flow:** Dashboard → "Connect" → xterm.js viewer → type `ls`, verify output
|
||||
- **Resize:** Browser window resize → PTY receives SIGWINCH → `htop` redraws correctly
|
||||
- **Session cleanup:** Close viewer → PTY process exits gracefully
|
||||
|
||||
### Manual Testing Scenarios
|
||||
|
||||
1. **Basic shell interaction:**
|
||||
- Connect to headless agent via dashboard
|
||||
- Type `ls -la`, verify colorized output
|
||||
- Run `vim test.txt`, verify cursor movement, editing, save/quit
|
||||
- Run `htop`, verify full-screen TUI app renders correctly
|
||||
|
||||
2. **Terminal resize:**
|
||||
- Start session at default 80x24
|
||||
- Resize browser window to 120x40
|
||||
- Run `tput cols; tput lines` → verify output matches
|
||||
- Run `htop` → verify UI scales to new dimensions
|
||||
|
||||
3. **Multi-line output:**
|
||||
- Run `dmesg | head -100` → verify scrollback works
|
||||
- Run `journalctl -f` → verify live log streaming
|
||||
|
||||
4. **Session recording playback:**
|
||||
- Perform session actions (ls, vim, htop)
|
||||
- End session
|
||||
- Admin dashboard → "View Recording" → verify asciicast playback
|
||||
|
||||
5. **Privilege escalation (sudo):**
|
||||
- Agent runs as `guruconnect` user (non-root)
|
||||
- Connect via terminal
|
||||
- Run `sudo apt update` → enter sudo password → verify command executes
|
||||
- Run `whoami` → verify shows `root` after sudo
|
||||
|
||||
### Performance
|
||||
|
||||
- **Latency target:** <100ms round-trip for input (same as GUI mode)
|
||||
- **Bandwidth:** ~1-5 KB/sec for typical terminal I/O (much lower than screen capture)
|
||||
- **Stress test:** Run `yes` command (infinite output) → verify relay doesn't OOM, rate-limit applied
|
||||
|
||||
## Effort Estimate & Dependencies
|
||||
|
||||
**Size:** Medium (5-7 weeks, 1 developer)
|
||||
|
||||
**Breakdown:**
|
||||
- Serial console implementation (Linux agent): 1.5 weeks
|
||||
- PTY implementation (Linux agent): 1.5 weeks
|
||||
- Mode selection + dispatcher: 0.5 weeks
|
||||
- Protobuf protocol updates (mode enum, console_mode flag): 0.5 weeks
|
||||
- Relay server terminal routing: 1 week
|
||||
- xterm.js web viewer integration: 1 week
|
||||
- Dashboard mode selector UI + routing: 0.5 weeks
|
||||
- Session recording + playback (both modes): 1 week
|
||||
- Testing (console + shell modes), edge cases, systemd integration: 1.5 weeks
|
||||
- Documentation (setup guide for serial console): 0.5 weeks
|
||||
|
||||
**Dependencies:**
|
||||
- **SPEC-010 Linux agent base** — PTY mode extends the Linux agent; can be implemented in parallel with SPEC-010's GUI capture
|
||||
- **xterm.js library** — mature, well-tested (used by VS Code, Jupyter, many commercial products)
|
||||
- **libc/nix crates** — standard Rust POSIX bindings
|
||||
- **SPEC-004 per-agent keys** — already shipped for persistent agent auth
|
||||
|
||||
**Unblocks:**
|
||||
- **Boot-level access** (GRUB menu, kernel parameters, single-user mode) via serial console mode
|
||||
- **Emergency recovery** (kernel panics, filesystem repair, systemd rescue shell) via serial console
|
||||
- **Server management** (Linux VMs, containers, bare metal) via shell mode
|
||||
- **SSH replacement** with centralized audit logging and GuruConnect auth
|
||||
- **Container debugging** (exec into running containers via GuruConnect)
|
||||
- **KVM-over-IP alternative** (serial console provides text-mode equivalent to IPMI Serial-over-LAN)
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Serial console permissions - root vs. capabilities?** — Opening `/dev/ttyS0` requires root. Options: (a) run agent as root for console mode, (b) use Linux capabilities (`CAP_SYS_TTY_CONFIG`), (c) add agent user to `dialout` group (may not work for `/dev/console`). Recommend (b) via systemd unit: `AmbientCapabilities=CAP_SYS_TTY_CONFIG`.
|
||||
|
||||
2. **Default mode if serial console unavailable?** — If `/dev/ttyS0` doesn't exist or permission denied, fall back to shell mode automatically or show error? Recommend auto-fallback with warning message in viewer.
|
||||
|
||||
3. **Serial console baud rate?** — v1 hardcodes 115200 (industry standard). Phase 2: make configurable if slower links needed (9600, 38400).
|
||||
|
||||
4. **Shell selection (PTY mode)?** — v1: `$SHELL` env var → `/bin/bash` → `/bin/sh`. Phase 2: dashboard setting to override shell per agent (`/bin/zsh`, `/bin/fish`).
|
||||
|
||||
5. **Concurrent sessions?** — v1: one console/shell session per agent connection (like SSH). Phase 2: tmux/screen integration for multi-viewer session sharing.
|
||||
|
||||
6. **Terminal recording format?** — Asciicast (JSON, industry standard, xterm.js playback support) vs. raw dump (more compact, custom playback). Recommend asciicast for v1.
|
||||
|
||||
7. **Command whitelisting (shell mode)?** — Optional Phase 2 feature. v1 is unrestricted shell access (same as SSH). Add `allowed_commands` array to agent config if compliance requires it.
|
||||
|
||||
8. **RBAC for console vs. shell access?** — Should some users only have shell access (not console, which grants boot-level control)? Recommend yes: add `console_access` permission, separate from `shell_access`.
|
||||
|
||||
9. **MFA for console mode?** — Given boot-level control risk, require MFA for console mode sessions? Defer to Phase 2 (MFA is a broader GuruConnect feature).
|
||||
|
||||
10. **Windows/macOS terminal mode?** — Defer. Windows Server typically uses RDP or SSH (OpenSSH built-in since Server 2019). macOS servers are rare. Linux headless servers are the primary use case.
|
||||
|
||||
11. **File upload/download via terminal?** — v1: use standard tools (`scp`, `rsync`, `wget`). Phase 2: integrate with SPEC (file transfer) for dashboard-native upload/download.
|
||||
|
||||
---
|
||||
|
||||
**Cross-references:**
|
||||
- SPEC-010: Cross-platform agents (macOS/Linux GUI) — headless mode extends Linux agent with PTY alternative
|
||||
- SPEC-004: Stable machine identity — headless agents use same deterministic `machine_uid` (`/etc/machine-id`)
|
||||
- ADR-001: GuruConnect is standalone — headless mode doesn't require GuruRMM integration
|
||||
- Future: File transfer spec (roadmap item) — will integrate with terminal mode for `scp`-like functionality
|
||||
Reference in New Issue
Block a user