Agenta

Interactive web-based environment for testing and iterating on prompts across multiple LLM providers (OpenAI, Anthropic, Ollama, LiteLLM) with automatic version tracking and configuration snapshots. Uses a FastAPI backend that manages prompt state, model selection, and parameter variations, while the Next.js frontend provides real-time prompt editing with side-by-side output comparison. Each variant is persisted as an immutable snapshot linked to an Application, enabling rollback and A/B testing workflows.

Executes parameterized evaluation workflows against testsets using a modular evaluator registry that supports both built-in evaluators (regex matching, LLM-as-judge, similarity scoring) and custom Python evaluators. The evaluation system uses a task queue pattern (via Celery or direct execution) to parallelize evaluator runs across test cases, with results aggregated into a comparison matrix. Evaluators are configured via JSON schema, allowing non-technical users to customize thresholds and prompts without code changes.

Provides a unified API gateway that abstracts differences between LLM providers (OpenAI, Anthropic, Ollama, Cohere, etc.) using the LiteLLM library. The proxy normalizes request/response formats, handles authentication with provider-specific keys, and computes token counts and costs automatically. This enables applications to switch between providers or use multiple providers without code changes. The proxy is deployed as a separate service and handles rate limiting, retries, and fallback logic.

Provides a web-based dashboard that visualizes evaluation results across variants, testsets, and time periods. The dashboard displays comparison matrices (variant × metric), aggregate statistics (mean, std dev, pass rate), and trend charts showing performance over time. Users can filter results by metadata (model, testset, date range) and export data for external analysis. The dashboard supports custom metric visualization and drill-down into individual test cases to understand failure modes.

Executes a prompt variant (application) against all test cases in a testset, collecting outputs and metrics. The system uses a task queue pattern to parallelize execution across test cases, with configurable concurrency limits to avoid rate limiting. Results are streamed to the frontend as they complete, providing real-time feedback. The system handles failures gracefully, retrying failed cases and collecting error logs for debugging. Execution results are persisted in the database and linked to the variant and testset for later analysis.

Provides a production-ready Docker Compose configuration for self-hosted deployment of the entire Agenta stack (frontend, backend, database, services). The deployment includes environment variable templates for configuring LLM providers, database connections, and authentication. Supports both OSS (open-source) and EE (enterprise edition) deployments with feature flags. Includes migration scripts for upgrading between versions without data loss.

Provides a unified LLM API proxy (via LiteLLM) that abstracts differences between LLM providers (OpenAI, Anthropic, Cohere, etc.) into a single interface. The proxy handles authentication, rate limiting, retry logic, and cost tracking across providers. Applications can switch between providers by changing a configuration parameter without code changes. Supports streaming responses and function calling across different provider APIs.

Provides a web-based annotation interface for human raters to score LLM outputs against testsets, with support for multiple annotation types (binary choice, multi-class, Likert scale, free-form feedback). The system tracks annotator identity, timestamps, and inter-rater agreement metrics (Cohen's kappa, Fleiss' kappa) to measure evaluation consistency. Annotations are stored in the backend database and can be compared against automated evaluation results to identify cases where human judgment diverges from metrics.

Manages collections of test cases (inputs, expected outputs, metadata) with version control and import/export capabilities. Testsets are stored as structured records in the backend database, supporting CSV/JSON import and export. The system tracks testset versions, allowing users to compare evaluation results across different testsets and identify performance regressions when testset coverage changes. Test cases can include dynamic variables that are substituted at evaluation time.

Enables side-by-side comparison of prompt variants or model configurations using evaluation results from the same testset. The system computes aggregate metrics (mean, median, std dev) for each variant and displays results in a comparison matrix. While the core comparison is deterministic, the framework supports filtering and slicing results by testset metadata to identify performance differences across subgroups. Results are persisted and can be exported for external statistical analysis.

Instruments LLM application execution with OpenTelemetry traces that capture request/response spans, token counts, latency, and cost. The system uses Python SDK decorators (@app, @step) to automatically wrap function calls and emit traces to a backend collector. Traces are stored in a time-series database and can be queried via the web UI to identify performance bottlenecks, cost drivers, and error patterns. Integration with LiteLLM proxy enables automatic token counting and cost calculation for LLM calls.

Provides a Python library (published as 'agenta' on PyPI) that enables developers to define LLM applications using decorators (@app, @step) that automatically register functions as variants and instrument them for tracing. The SDK handles parameter serialization, testset execution, and result collection without requiring explicit API calls. Applications are defined as Python functions with type-annotated parameters, which are automatically exposed in the web UI as configurable inputs. The SDK supports both synchronous and asynchronous execution.

Provides secure storage for API keys and sensitive configuration values (e.g., LLM provider keys, database credentials) with automatic injection into application execution contexts. Secrets are encrypted at rest in the backend database and decrypted only when needed for execution. The system supports both global secrets (shared across workspace) and application-specific secrets. Secrets are never exposed in the web UI or logs; only secret names are visible to users.

Implements organization and workspace hierarchies with role-based access control (RBAC) to isolate data and functionality across teams. Each workspace has its own applications, testsets, evaluations, and secrets. Users are assigned roles (admin, editor, viewer) that determine which operations they can perform. The system enforces access control at the API level, preventing unauthorized access to workspace data. Authentication is handled via OIDC, SAML, or local accounts.

Provides production-ready Docker Compose configuration that orchestrates all Agenta services (web frontend, FastAPI backend, PostgreSQL database, OpenTelemetry collector, LiteLLM proxy) with a single command. Configuration is managed via environment variables (.env file), enabling users to customize deployment without modifying Docker Compose files. The setup includes health checks, volume mounts for persistence, and networking configuration. SSL/TLS support is available via reverse proxy configuration.