Keywords AI

Routes requests to 500+ external LLM models (OpenAI, Anthropic, etc.) through a single API endpoint, abstracting provider-specific request/response formats and handling protocol translation. Implements request caching, automatic retries with exponential backoff, and fallback routing to alternative models when primary provider fails, reducing integration complexity from managing N provider SDKs to a single gateway interface.

Stores, versions, and deploys prompts through a web IDE with git-like version control, enabling teams to track prompt changes, rollback to previous versions, and deploy new prompts to production through the gateway without code changes. Integrates with the unified gateway to serve deployed prompt versions at inference time, supporting A/B testing by routing traffic to different prompt versions.

Enables A/B testing by deploying multiple prompt or model versions and routing traffic to each variant based on configurable split percentages (e.g., 50% to variant A, 50% to variant B). Automatically collects metrics for each variant (latency, cost, quality) and provides statistical comparison dashboards to determine which variant performs better. Supports gradual rollout (canary deployment) by starting with small traffic percentages and increasing based on performance.

Supports multiple team members with role-based access control (RBAC), enabling organizations to grant different permissions to engineers, product managers, and finance teams. Tracks who made changes to prompts, deployments, and alert configurations with audit logs, and supports team-scoped dashboards and alerts. Integrates with Google SSO for authentication (Pro/Team tiers) with SAML support on Enterprise tier.

Caches identical LLM requests at the gateway level and returns cached responses without calling the LLM provider, reducing latency and cost for repeated queries. Supports cache invalidation strategies (TTL, manual) and provides cache hit/miss metrics on dashboards. Works transparently for requests routed through the Respan gateway without application-level changes.

Offers self-hosted deployment option for Enterprise tier customers, allowing Keywords AI infrastructure to run on customer's own servers or cloud account. Enables data residency compliance (e.g., data must stay in EU for GDPR). Self-hosted deployment includes all Keywords AI features (gateway, tracing, evaluation, dashboards). Requires customer to manage infrastructure, updates, and security patches. Specific deployment options (Kubernetes, Docker, VMs) not documented.

Supports SAML 2.0 authentication for Enterprise tier customers, enabling integration with corporate identity providers (Okta, Azure AD, etc.). Allows centralized user management and access control through existing identity infrastructure. Supports role-based access control (RBAC) and single sign-on (SSO). SAML is available only on Enterprise tier; Pro/Team tiers use Google OAuth.

Captures complete execution traces from production LLM calls including request/response content, latency, token counts, cost, and custom metadata, storing traces in a searchable index with 7-30 day retention. Enables filtering and searching by content keywords, latency ranges, cost thresholds, quality tags, and custom properties, with trace replay functionality allowing developers to re-run requests through the playground for debugging.

Evaluates LLM outputs using three judge types—code-based (custom Python functions), human review (manual annotation), and LLM-as-judge (using another LLM to score outputs)—against versioned evaluation datasets. Stores evaluation scores in a queryable database, enabling teams to track quality metrics over time, compare model/prompt versions, and identify regressions. Supports custom evaluation metrics and integrates with dashboards for visualization.

Monitors production LLM metrics (latency, cost, quality, error rate) in real-time and triggers alerts via Slack, email, or SMS when thresholds are breached. Supports conditional alerting based on custom properties (e.g., alert only for requests from specific users or with specific tags) and can trigger automated workflows or webhooks in response to production signals, enabling teams to respond to issues without manual monitoring.

Provides a visual dashboard builder with 80+ pre-built graph types for tracking quality, latency, cost, and behavior metrics across LLM requests. Supports custom properties and dimensions, enabling teams to slice metrics by model, prompt version, user segment, or any custom tag. Dashboards update in real-time as new requests are processed, and can be shared across teams for collaborative monitoring.

Exports production traces and evaluation scores in batch format (JSONL, CSV) on-demand or on a schedule via webhooks, enabling teams to integrate Respan data into data warehouses, analytics platforms, or custom analysis pipelines. Supports conditional export (e.g., export only traces matching specific filters) and PII masking for compliance, with configurable retention policies to manage data storage costs.

Accepts trace data in OpenTelemetry format, enabling teams to send LLM execution traces from their own instrumentation rather than routing all requests through Respan gateway. Integrates with OpenTelemetry collectors and exporters, allowing teams to use Respan as a backend for observability data collected from distributed systems. Supports custom span attributes and semantic conventions for LLM-specific metadata.

Tracks LLM API costs at request granularity (cost per token, per request, per model) by integrating with provider pricing data, aggregates costs by model/prompt/user/custom dimension, and enables budget alerts when spending exceeds thresholds. Provides cost breakdown dashboards showing which models, prompts, or user segments are driving expenses, enabling teams to optimize for cost without sacrificing quality.

Sends real-time alerts and notifications to Slack channels when production thresholds are breached (latency, cost, quality, error rate), with rich formatting including metric values, affected requests, and recommended actions. Supports channel routing based on alert type or custom properties, enabling teams to direct different alerts to different channels (e.g., cost alerts to finance, quality alerts to ML team).