Cua

Exposes the Cua ComputerAgent framework as an MCP (Model Context Protocol) server, enabling Claude Desktop and other MCP clients to invoke computer-use capabilities through standardized tool calling. The MCP server translates incoming tool calls into ComputerAgent method invocations, manages screenshot capture and action execution state, and returns structured responses back through the MCP protocol, eliminating the need for direct SDK integration.

Implements a unified agent loop that abstracts 100+ vision-language models (Claude, GPT-4V, Gemini, open-source models via Ollama) behind a single ComputerAgent interface. The loop captures screenshots, formats them with task context using the Responses API message format, sends them to the selected VLM, parses structured action responses, and executes OS-level operations. Model selection is decoupled from agent logic through a provider architecture, enabling runtime model switching without code changes.

Exposes agent execution capabilities via HTTP REST API and WebSocket connections, enabling remote clients to trigger agent runs and stream results in real-time. The server is built on FastAPI and handles authentication, request validation, and response serialization. Clients can submit tasks, poll for status, retrieve trajectories, and stream screenshots/actions via WebSocket. The server supports multiple concurrent agent executions with per-request isolation. OS-specific handlers are abstracted, allowing the server to run on any platform and target any execution environment.

Implements the Anthropic Responses API message format for structured agent reasoning and action specification. This format enables models to return structured actions (click, type, scroll) with explicit reasoning, reducing parsing ambiguity and improving reliability. The framework automatically converts model responses in this format into executable actions, handling validation and error recovery. Support for Responses API is built into the agent loop, with fallback to text parsing for models that don't support structured output.

Provides comprehensive telemetry and observability through structured logging, metrics collection, and integration with observability platforms. The system logs all agent loop steps (screenshot, reasoning, action, result) with timestamps, model outputs, and error details. Metrics include latency per step, token usage, cost, and success rates. Logs are structured (JSON) for easy parsing and can be exported to external systems (CloudWatch, Datadog, Prometheus). The telemetry system is pluggable, allowing custom exporters to be registered.

Abstracts execution environments (Docker containers, Lume macOS VMs, Windows Sandbox, host OS) behind a unified provider interface, allowing agents to target different execution contexts without code changes. The provider architecture handles environment-specific screenshot capture (X11/Wayland on Linux, native APIs on macOS/Windows), action execution (xdotool, native APIs), and resource lifecycle management. Agents specify target environment at runtime; the framework routes screenshot and action calls to the appropriate provider implementation.

Captures screenshots from the target environment and optionally augments them with semantic object mapping (SOM) — overlaying bounding boxes and labels for interactive UI elements (buttons, inputs, links). The SOM system uses vision models to identify clickable regions and assigns them numeric IDs, enabling agents to reference UI elements by semantic identity rather than pixel coordinates. This reduces hallucination and improves action accuracy, especially for complex interfaces. SOM generation is optional and configurable per agent run.

Translates high-level action specifications (click, type, scroll, wait) into OS-specific commands executed on the target environment. The framework implements native handlers for Linux (xdotool, X11/Wayland), macOS (native APIs), and Windows (pyautogui, native APIs), abstracting platform differences. Actions are queued, executed sequentially, and validated; failures trigger retry logic or error reporting. The action execution layer is decoupled from agent reasoning, allowing custom action handlers to be plugged in.

Provides extension points for customizing the agent loop without modifying core framework code. Developers can implement custom agent loops by subclassing the base loop, overriding specific methods (e.g., screenshot capture, action parsing, reasoning), and registering callbacks at key points (pre/post screenshot, pre/post action, loop completion). The callback system enables monitoring, logging, cost tracking, and conditional loop termination. Custom tools can be registered and made available to agents through a tool registry.

Tracks API costs and token usage across agent executions, with per-model cost calculation based on input/output token counts and model-specific pricing. The system maintains a budget limit and can terminate agents when budget is exceeded. Cost tracking is integrated into the agent loop via callbacks, enabling real-time cost monitoring and reporting. Supports multiple cost backends (OpenAI, Anthropic, custom) and generates cost reports by model, task, and time period.

Records complete agent execution trajectories (screenshots, actions, reasoning, errors) to disk or cloud storage, enabling post-execution analysis, debugging, and evaluation. Trajectories include timestamps, model outputs, action results, and environment state at each step. The system supports trajectory replay — re-executing recorded actions against a fresh environment to validate reproducibility or test modifications. Trajectories can be exported in standard formats (JSON, video) for sharing and analysis.

Integrates with OSWorld benchmark suite and supports custom evaluation workflows for measuring agent performance. The evaluation system runs agents against predefined tasks, collects trajectories, and computes metrics (success rate, step efficiency, cost per task). Results are compared against baseline models and can be visualized in dashboards. The framework supports both automated evaluation (batch runs) and interactive evaluation (human-in-the-loop validation). Custom evaluation metrics can be implemented and registered.

Manages macOS virtual machines via the Lume platform, enabling agents to run against macOS environments without requiring physical hardware. The system handles VM provisioning, lifecycle management (start, stop, snapshot), and image caching. Agents can target specific macOS versions and software configurations by selecting pre-built VM images. The Lume provider abstracts VM communication details, presenting a uniform interface to the agent loop. Supports concurrent VM execution for parallel testing.