UFO

UFO² captures Windows desktop screenshots, annotates UI controls with bounding boxes and accessibility metadata, and uses LLM reasoning to decompose natural language tasks into sequences of UI interactions (clicks, text input, keyboard commands). The Host Agent orchestrates high-level task planning while App Agents execute granular actions within specific applications, maintaining state machines to track progress and handle failures across multi-step workflows.

UFO² captures full desktop screenshots and overlays bounding boxes with unique IDs for every interactive UI control (buttons, text fields, dropdowns, etc.) extracted via Windows Accessibility API (UIA) and COM object inspection. Annotations include control type, label, state, and accessibility properties, creating a structured representation of the UI that LLMs can reason about without OCR. The system handles dynamic UI updates by re-capturing and re-annotating on each agent round.

UFO² abstracts LLM interactions behind a provider-agnostic interface supporting OpenAI, Anthropic, Azure OpenAI, and local Ollama models. The system handles provider-specific details (API authentication, request formatting, response parsing) transparently. For structured outputs, UFO² uses JSON schema validation and function calling APIs (where available) to ensure agents produce well-formed action specifications. Supports custom model integration via a plugin interface.

UFO² uses YAML/JSON configuration files to define agent behavior, LLM settings, tool definitions, and deployment modes without code changes. Configuration includes agent type (Host/App), LLM provider and model, prompt templates, tool definitions, knowledge base paths, and deployment mode (local, service, or Galaxy). The system loads configurations at startup and applies them consistently across all agent instances, enabling rapid experimentation and deployment variations.

UFO³ Galaxy Framework includes a web-based UI for monitoring and controlling multi-device automation. The UI displays registered devices, running tasks, execution traces, and device health metrics. Users can submit new tasks, view real-time execution progress (including screenshots from remote devices), inspect action history, and manage device lifecycle (register, deregister, restart). The UI communicates with the Galaxy controller via REST APIs or WebSockets for real-time updates.

UFO² agents implement explicit state machines defining valid state transitions (e.g., Idle → Planning → Executing → Observing → Idle). Each agent round transitions through states, with state-specific logic for handling errors, retries, and recovery. If an action fails, the agent can retry within the same Round, escalate to the Host Agent, or transition to an error recovery state. State machines enable deterministic behavior, clear error handling, and recovery strategies without ad-hoc exception handling.

UFO² implements a two-tier agent hierarchy where the Host Agent receives natural language tasks, decomposes them into sub-tasks, and delegates execution to specialized App Agents running within specific application contexts. Each App Agent maintains its own state machine, action history, and application-specific knowledge, communicating results back to the Host Agent. The Host Agent orchestrates task flow, handles inter-application dependencies, and decides when to switch between App Agents or retry failed sub-tasks.

UFO² organizes execution into Sessions (long-lived contexts for a task) and Rounds (individual agent decision cycles). Each Round captures the current UI state (screenshot + annotations), executes one or more actions, observes results, and feeds observations back to the agent for the next Round. Sessions maintain action history, context windows, and error recovery state across multiple Rounds, enabling agents to learn from previous attempts and adapt strategies.

UFO² supports two action types: LLM-reasoned actions (click, type, keyboard shortcuts decided by the agent) and deterministic COM automation actions (direct method calls to application objects via Windows COM interfaces). The system intelligently routes actions based on precision requirements—using COM for exact operations (e.g., setting cell values in Excel) and LLM reasoning for exploratory tasks (e.g., finding a button in an unfamiliar UI). Hybrid execution reduces LLM latency for well-defined operations while maintaining flexibility for novel scenarios.

UFO² integrates with the Model Context Protocol (MCP) to expose external tools and services as callable functions within agent reasoning. Agents can invoke MCP servers (local or remote) to access capabilities like web search, file operations, database queries, or custom business logic. The system provides a framework for creating custom MCP servers that wrap application-specific operations, enabling agents to extend their capabilities beyond UI automation.

UFO³ Galaxy Framework extends UFO² to orchestrate tasks across multiple Windows devices. A Constellation Agent receives high-level tasks, decomposes them into device-specific sub-tasks, and distributes execution to UFO² agents running on remote devices via the Agent Interaction Protocol (AIP). The system manages device registration, task routing, result aggregation, and failure recovery across heterogeneous device fleets, enabling workflows that span multiple machines (e.g., data collection on Device A, processing on Device B, reporting on Device C).

UFO³ implements the Agent Interaction Protocol (AIP), a structured communication protocol enabling UFO² agents on remote devices to register with the Galaxy controller, receive task assignments, report execution status, and stream results back. AIP defines message formats for task requests, action execution, observation reporting, and error handling, abstracting away network transport details. The protocol supports both synchronous (request-response) and asynchronous (streaming) communication patterns.

UFO² integrates a Retrieval-Augmented Generation (RAG) system that stores domain knowledge (application documentation, automation patterns, troubleshooting guides) in a vector database. When agents encounter novel situations, they query the knowledge base to retrieve relevant context, which is injected into the LLM prompt. The system supports multiple vector database backends (Chroma, Weaviate, Pinecone) and provides tools for creating, updating, and managing knowledge documents.

UFO² implements a sophisticated prompt construction system that assembles multi-modal context (screenshots, UI annotations, action history, retrieved knowledge, task description) into structured prompts for LLM reasoning. The system manages prompt components (system instructions, task context, observation history, tool definitions) and applies strategies like context pruning, summarization, and priority-based truncation to fit within LLM token limits. Supports multi-modal prompts combining text, images, and structured data.