opentelemetry-based application instrumentation with decorator-driven span generation
Wraps LLM application methods using the @instrument decorator to automatically generate structured OpenTelemetry spans (RECORD_ROOT, GENERATION, RETRIEVAL, EVAL) without modifying core application logic. The decorator integrates with a TracerProvider that captures execution context, method inputs/outputs, and timing metadata, then exports spans to configured backends (SQLite, PostgreSQL, Snowflake). This enables zero-friction observability for framework-agnostic applications.
Unique: Uses a decorator-based instrumentation model that generates structured OTEL spans with semantic span kinds (GENERATION, RETRIEVAL, EVAL) specific to LLM workflows, rather than generic HTTP/RPC spans. Integrates directly with TruSession for unified span collection and evaluation lifecycle management.
vs alternatives: Simpler than manual OTEL instrumentation and more LLM-aware than generic APM tools; requires less boilerplate than Langsmith's tracing while maintaining OTEL standard compliance.
llm-based feedback function evaluation with multi-provider support
Computes evaluation metrics (groundedness, relevance, coherence, custom metrics) by executing feedback functions that call LLM APIs with structured prompts. The Feedback class defines metric logic; LLMProvider interface abstracts over OpenAI, Bedrock, Cortex, HuggingFace, and LiteLLM endpoints. Evaluation runs asynchronously via a background Evaluator thread, storing results linked to application spans. Supports both synchronous (blocking) and deferred (async) evaluation modes.
Unique: Abstracts LLM provider selection behind LLMProvider interface, enabling same feedback function to run against OpenAI, Bedrock, Cortex, or local models without code changes. Integrates evaluation lifecycle with span collection via RunManager, enabling automatic metric computation on application traces.
vs alternatives: More flexible than Langsmith's built-in metrics (supports custom LLM providers and deferred evaluation); more integrated than standalone evaluation frameworks (metrics tied directly to application spans and session lifecycle).
snowflake event table export and server-side evaluation pipeline
Exports OTEL spans directly to Snowflake event tables for server-side querying and analysis. SnowflakeEventTableDB connector implements DBConnector interface, batching span exports asynchronously. Enables server-side evaluation pipeline where feedback functions execute in Snowflake Cortex (LLM provider) rather than client-side, reducing data transfer and enabling SQL-based metric computation. Integrates with Snowflake's native OTEL support.
Unique: Exports OTEL spans directly to Snowflake event tables and enables server-side evaluation in Snowflake Cortex, avoiding data export and enabling native SQL querying. Tighter integration than generic OTEL exporters.
vs alternatives: More efficient than client-side evaluation for large-scale deployments; enables SQL-based analytics on trace data within data warehouse.
run management and external agent integration for distributed evaluation
RunManager class orchestrates application runs, tracking run metadata (ID, timestamp, app name, version), linking spans and metrics to runs, and managing run lifecycle. Supports external agent integration for distributed evaluation — agents can retrieve pending runs, execute feedback functions, and report results back to central database. Enables horizontal scaling of evaluation workload across multiple workers.
Unique: Provides RunManager for tracking run lifecycle and metadata, with support for external agents to execute distributed evaluation. Enables horizontal scaling of evaluation workload.
vs alternatives: More integrated than generic job queues; provides run-level abstraction specific to LLM evaluation workflows.
backwards compatibility layer for trulens_eval<1.0.0 api migration
This package (trulens-eval) provides backwards-compatible API for applications built against trulens_eval<1.0.0, mapping old API calls to new trulens-core>=1.0.0 implementations. Enables existing applications to upgrade without code changes. Acts as compatibility shim during migration period, allowing gradual adoption of new API.
Unique: Provides compatibility shim mapping trulens_eval<1.0.0 API to trulens-core>=1.0.0 implementations, enabling zero-change upgrades for existing applications.
vs alternatives: Enables gradual migration path vs requiring immediate rewrite; reduces upgrade friction for existing users.
session-based application lifecycle and database connection management
TruSession class provides centralized orchestration for database connections, OTEL setup, evaluation scheduling, and run lifecycle. Manages DBConnector abstraction (SQLAlchemy, Snowflake event tables) for span/metric persistence, coordinates Evaluator thread for async feedback execution, and maintains context across application invocations. Session acts as entry point for developers: initialize once, wrap application, retrieve results.
Unique: Centralizes database, OTEL, and evaluation orchestration in single TruSession object that manages DBConnector abstraction, Evaluator thread lifecycle, and run context. Enables context manager pattern (with statement) for automatic resource cleanup.
vs alternatives: Simpler than manual OTEL setup and database connection management; more integrated than standalone database libraries because it couples persistence with evaluation scheduling and span collection.
multi-backend persistence with database abstraction layer
DBConnector interface abstracts storage backend selection (SQLAlchemy for SQLite/PostgreSQL/MySQL, SnowflakeEventTableDB for Snowflake). Stores spans, feedback metrics, and run metadata in normalized schema. SQLAlchemy backend uses ORM models for relational storage; Snowflake backend exports OTEL spans directly to event tables for server-side querying. Enables schema migrations and versioning for database evolution.
Unique: Provides DBConnector abstraction that supports both relational (SQLAlchemy) and cloud-native (Snowflake event tables) backends with unified API. Snowflake backend exports OTEL spans directly to event tables, enabling server-side querying without ETL.
vs alternatives: More flexible than single-backend solutions; Snowflake integration is deeper than generic OTEL exporters because it uses event table schema optimized for trace data.
framework-specific application wrapping with semantic span kinds
Provides framework-specific wrapper classes (TruChain for LangChain, TruGraph for LangGraph, TruLlama for LlamaIndex, TruBasicApp/TruCustomApp for custom apps) that intercept application execution and generate semantically-typed spans (GENERATION for LLM calls, RETRIEVAL for vector search, EVAL for feedback). Wrappers preserve original framework APIs while injecting instrumentation transparently.
Unique: Provides framework-specific wrappers that generate semantically-typed spans (GENERATION, RETRIEVAL, EVAL) tailored to LLM workflows, rather than generic function call spans. Wrappers intercept framework-level operations (LLM calls, vector search) to assign correct span kinds automatically.
vs alternatives: More semantic than generic OTEL instrumentation; more framework-aware than manual span creation; preserves original framework APIs unlike some observability solutions that require code rewriting.
+5 more capabilities