OpenAgents

Provides a single Next.js-based web UI that routes user queries to specialized agent implementations (Data, Plugins, Web) through a Flask backend, managing agent selection, state transitions, and real-time streaming responses. The system uses a service-oriented architecture where each agent type is independently deployable but communicates through standardized API endpoints, enabling users to switch between agents within a single conversation context without manual reconfiguration.

Executes Python and SQL code in an isolated environment to perform data manipulation, transformation, and visualization tasks. The Data Agent accepts structured inputs (CSV, JSON, Excel), parses them into pandas DataFrames, executes user-requested operations through a restricted Python/SQL interpreter, and returns results as visualizations, tables, or raw data. This capability integrates with the backend's memory system to cache intermediate results and maintain execution context across multiple queries.

Provides a framework for developers to create custom agent types by implementing a standard agent interface (inherited from a base Agent class) and registering them with the backend. Custom agents can leverage shared adapters (memory, streaming, callbacks) and integrate with the existing UI without modification. The system uses a plugin discovery mechanism to load agents from the agents/ directory, enabling drop-in extensibility.

Provides Docker Compose configuration for deploying OpenAgents as containerized services (frontend, backend, MongoDB, Redis) with environment variable-based configuration. The system supports both local development (docker-compose up) and production deployments with proper networking, volume management, and service dependencies. Configuration is externalized through .env files, enabling easy switching between LLM providers, database backends, and deployment targets.

Provides access to 200+ third-party plugins (shopping, weather, scientific tools, etc.) through a plugin registry and automatic selection mechanism. The Plugins Agent uses the LLM to determine which plugins are relevant to a user query, constructs appropriate API calls with parameter binding, and aggregates results. The system maintains a plugin manifest with schemas, descriptions, and authentication requirements, enabling the agent to reason about tool availability without manual configuration per query.

Enables agents to autonomously navigate websites, extract information, and interact with web pages through a Chrome extension that captures page state and DOM interactions. The Web Agent receives high-level instructions (e.g., 'find the cheapest flight'), translates them into browser actions (click, scroll, fill form), and uses vision/OCR capabilities to interpret page content. The extension maintains a session context and screenshot history, allowing the agent to reason about page state changes and plan multi-step navigation sequences.

Manages conversation history, user context, and agent state across sessions using MongoDB as the primary store and Redis for caching frequently accessed data. The system stores messages, execution results, file uploads, and agent-specific state in structured collections, enabling users to resume conversations, reference past interactions, and maintain context across multiple agent switches. Memory is indexed by conversation ID and user ID, with TTL policies for automatic cleanup of old sessions.

Abstracts interactions with multiple LLM providers (OpenAI, Anthropic, local models via Ollama) through a unified interface, handling API key management, request formatting, streaming response parsing, and error handling. The system maintains provider-specific adapters that translate between OpenAgents' internal message format and each provider's API schema, enabling users to swap LLM backends without changing agent code. Configuration is environment-based, allowing runtime provider selection.

Implements server-sent events (SSE) and WebSocket-based streaming to deliver agent responses to the frontend in real-time, enabling users to see intermediate results, code execution progress, and tool calls as they happen. The backend streams tokens from the LLM, execution logs from the Data Agent, and plugin results as they complete, with the frontend rendering updates incrementally. This avoids blocking on long-running operations and provides perceived performance improvements.

Translates natural language queries into executable operations (Python code, SQL queries, API calls, browser actions) by using the LLM to understand intent and generate appropriate code or commands. The system maintains a library of operation templates and examples, uses few-shot prompting to guide code generation, and validates generated code before execution. This enables users to express complex data operations, plugin calls, and web interactions in plain English without learning syntax.

Handles file uploads (CSV, JSON, Excel) through the web UI, automatically detects file format and schema, parses data into structured representations (pandas DataFrames), and stores metadata in MongoDB for later reference. The system validates file size, checks for encoding issues, and provides users with a preview of parsed data before analysis. Uploaded files are cached in Redis for quick access across multiple queries.

Maintains isolated state for each agent type (Data Agent execution context, Plugins Agent tool registry, Web Agent session state) while sharing common context (conversation history, user preferences) through a unified backend. Each agent has its own state store in MongoDB, with adapters that translate between agent-specific formats and the common interface. This enables agents to maintain specialized context (e.g., Data Agent's DataFrame cache) without polluting shared state.