agentops

Records complete execution traces of AI agent runs including LLM calls, tool invocations, and state transitions. Implements automatic instrumentation via Python decorators and context managers that capture function calls, arguments, return values, and timing metadata without requiring manual logging code. Stores traces in a session-based structure enabling replay and debugging of multi-step agent workflows.

Automatically intercepts and logs all LLM API calls (prompts, completions, token counts, latency) across multiple providers. Implements provider-agnostic instrumentation that wraps OpenAI, Anthropic, Cohere, and other client libraries to capture request/response metadata. Aggregates usage metrics and calculates per-call and per-session costs based on published pricing models.

Records all agent actions in an immutable audit log suitable for compliance and regulatory requirements. Implements tamper-evident logging with checksums and timestamps. Provides filtering and export capabilities for compliance reporting (HIPAA, SOC2, etc.) and enables retention policies based on data sensitivity.

Captures all tool/function invocations made by agents including function name, arguments, return values, and execution time. Implements automatic wrapping of tool registries and function definitions to log calls without modifying tool implementations. Validates tool schemas and can enforce constraints like argument types, return value formats, and execution timeouts.

Captures periodic snapshots of agent internal state including memory, context windows, and decision variables throughout execution. Implements state serialization that preserves complex Python objects (lists, dicts, custom classes) and stores them alongside execution traces. Enables comparison of state across execution steps to identify where agent behavior diverged from expected paths.

Provides a web-based UI for viewing recorded agent sessions with interactive timeline visualization, LLM call details, tool invocation logs, and cost breakdowns. Implements client-side rendering of execution traces with filtering and search capabilities. Supports session replay mode that reconstructs agent execution step-by-step with state snapshots and decision points highlighted.

Tracks interactions between multiple agents in a system including message passing, shared state updates, and coordination events. Implements correlation of traces across agent instances using unique session IDs and parent-child relationships. Visualizes agent communication patterns and identifies bottlenecks or deadlocks in multi-agent workflows.

Analyzes execution traces to identify performance bottlenecks including slow LLM calls, expensive tool invocations, and inefficient agent loops. Implements statistical analysis of timing data to flag outliers and suggests optimization opportunities. Compares performance across multiple sessions to identify regressions or improvements.

Captures exceptions, API errors, and agent failures with full context including the execution state at failure time. Implements error grouping that clusters similar failures across sessions to identify recurring issues. Provides root cause analysis by correlating errors with preceding LLM calls, tool invocations, and state changes.

Provides automatic instrumentation for OpenAI, Anthropic, Cohere, and other LLM provider Python SDKs through monkey-patching or wrapper classes. Implements provider-specific request/response parsing to extract prompts, completions, and metadata without modifying user code. Maintains compatibility with provider SDK updates through version detection and conditional instrumentation.

Provides structured logging API that automatically includes execution context (session ID, agent ID, step number) in all log messages. Implements context managers and decorators that propagate context through function calls and async operations. Integrates with Python's logging module to enable filtering and routing based on context.