Omi – watches your screen, hears conversations, tells you what to do

Continuously captures the active window or full screen at configurable intervals, processes frames through vision models (likely Claude Vision or similar), and extracts semantic understanding of UI state, text content, and visual context. Uses frame buffering and differential analysis to avoid redundant processing of unchanged screens, enabling efficient monitoring of user activity without overwhelming the inference pipeline.

Captures ambient audio from the device microphone in real-time, streams it to a speech-to-text engine (likely Whisper or similar), and converts spoken words into structured text with speaker identification when possible. Implements audio buffering and VAD (voice activity detection) to avoid processing silence, reducing API calls and latency. Maintains a rolling transcript window for context in subsequent reasoning steps.

Fuses real-time screen captures, audio transcripts, and user interaction history into a unified context representation that the reasoning engine can query. Implements a sliding-window memory buffer (likely 5-30 minutes of recent context) with semantic indexing to enable efficient retrieval of relevant past states. Uses embeddings or keyword matching to surface contextually relevant information when the agent needs to reason about what the user is doing.

Analyzes aggregated context (screen state + transcript + history) through a reasoning model (likely Claude or GPT-4) to infer the user's current intent and recommend proactive actions. Uses chain-of-thought prompting to decompose the user's situation into actionable steps, then ranks recommendations by relevance and confidence. Implements a feedback loop where user acceptance/rejection of recommendations trains the ranking model.

Translates recommended actions into executable operations by mapping them to available tools (calendar APIs, email clients, code editors, web browsers, etc.). Implements a function-calling interface where the reasoning model can request tool execution with parameters, then executes those requests through OS-level automation (likely AppleScript on macOS, PowerShell on Windows, or D-Bus on Linux) or direct API calls. Includes safety checks to prevent unintended side effects (e.g., confirming before sending emails).

Implements configurable data retention policies that control how long screen captures, audio transcripts, and context are stored locally before deletion. Supports optional local processing of sensitive operations (e.g., running Whisper locally instead of sending audio to the cloud) to minimize data transmission. Includes audit logging to track what data was captured, processed, and deleted, enabling compliance with privacy regulations.

Collects explicit user feedback (thumbs up/down, corrections, rejections) on agent recommendations and uses this to refine future suggestions. Implements a lightweight preference model that tracks which types of recommendations the user accepts or rejects, enabling personalization without requiring full model retraining. Stores preferences locally and uses them to re-rank recommendations before presenting them to the user.

Abstracts OS-specific screen capture and audio APIs (macOS: AVFoundation/ScreenCaptureKit, Windows: DXGI/Windows.Media.Capture, Linux: X11/Wayland/PulseAudio) behind a unified interface, enabling the agent to work consistently across platforms. Handles platform-specific permissions, frame rate negotiation, and audio format conversion automatically. Implements fallback mechanisms for unsupported configurations (e.g., Wayland on Linux).

Tracks CPU, memory, and API usage in real-time, implementing adaptive throttling to prevent resource exhaustion. Monitors inference latency and adjusts capture frequency or context window size dynamically to maintain responsiveness. Implements metrics collection (frame processing time, API call latency, token consumption) for debugging and optimization. Provides dashboards or CLI output showing resource usage and performance bottlenecks.

Provides a plugin interface that allows developers to register custom tools, integrations, and reasoning modules without modifying core code. Implements a discovery mechanism (likely directory scanning or manifest-based) to load plugins at startup, and a standardized interface (function signature, input/output schema) for plugins to expose capabilities. Supports plugins written in Python or via REST APIs, enabling integration with external services and custom business logic.