Galileo

Ingests structured execution traces from deployed LLM applications capturing models, prompts, function calls, context, and metadata in a unified schema. Processes traces through a centralized observability pipeline that correlates signals across the full execution path, enabling step-by-step workflow reconstruction and failure attribution. Supports ingestion via REST API, MCP server, and SDK integrations with configurable sampling and filtering at ingest time.

Analyzes model outputs against provided context and ground truth to identify factual inconsistencies, unsupported claims, and fabricated information. Uses a combination of LLM-as-judge evaluation and Luna distilled models to detect when generated text contradicts source documents or makes claims without supporting evidence. Operates on trace data post-inference, enabling both real-time guardrails and offline batch analysis of historical outputs.

Enables configurable sampling and filtering of traces at ingest time to reduce trace volume and associated costs. Supports filtering by criteria (e.g., only failures, high-latency requests) and sampling strategies (e.g., 10% of all traces, 100% of failures). Filtered traces are excluded from trace count limits but can still be analyzed if stored.

Supports evaluation of outputs from any LLM provider (OpenAI, Anthropic, open-source models, etc.) using the same metric library and guardrails. Metrics are provider-agnostic and can be applied to any model output regardless of source. Enables comparison of outputs from different providers using consistent evaluation criteria.

Tracks evaluation metrics over time and automatically detects regressions (quality drops) in model outputs. Compares current metric values against historical baselines and alerts when metrics fall below configured thresholds. Supports trend visualization and statistical significance testing to distinguish real regressions from noise.

Provides 20+ out-of-box evaluation metrics pre-configured for common LLM use cases (RAG, agents, safety, security) that automatically score model outputs against configurable criteria. Metrics are implemented as Luna distilled models that run at 97% lower cost than LLM-as-judge alternatives. Metrics can be applied to historical traces, new inferences, or custom datasets without code changes, with results aggregated into dashboards and reports.

Enables users to define custom evaluation metrics using a domain-specific language or configuration interface, then automatically apply them to traces and datasets. Custom metrics integrate into CI/CD pipelines as quality gates that block deployments if metrics fall below configured thresholds. Metrics are versioned and can be tested against historical traces before deployment, with results tracked over time to identify regressions.

Analyzes multi-step agent execution traces to identify failure patterns, incorrect tool selection, and suboptimal decision-making. Detects specific failure modes (e.g., 'hallucination caused incorrect tool inputs') by correlating agent actions with outcomes. Provides prescriptive debugging suggestions (e.g., 'Best action: Add few-shot examples') based on pattern analysis. Failure detection is quantified with percentage metrics (e.g., '15% Failure Detected') aggregated across trace populations.

Ingests datasets (synthetic, development, or production) with ground truth labels and runs evaluation metrics against model outputs to measure quality. Supports batch evaluation of historical data and continuous evaluation of new inferences against the same dataset. Results are aggregated into quality metrics and trend reports, enabling data-centric debugging by identifying which data characteristics correlate with failures.

Distills high-quality evaluation logic from expensive LLM judges (GPT-4o, Claude) into proprietary Luna models that run at 97% lower cost while maintaining evaluation accuracy. Luna models are pre-trained on evaluation tasks and deployed as low-latency inference endpoints. Users can apply Luna models to any evaluation task (hallucination detection, metric scoring, guardrail enforcement) without managing separate inference infrastructure.

Deploys Luna distilled models as real-time guardrails that evaluate model outputs during inference and block or flag unsafe/low-quality responses before they reach users. Guardrails run on Galileo's low-latency dedicated inference servers (Enterprise tier) or can be integrated into application inference pipelines. Supports multiple guardrail types (safety, security, quality) with configurable thresholds and actions (block, flag, modify).

Analyzes trace patterns and failure modes to generate prescriptive debugging suggestions (e.g., 'Add few-shot examples', 'Improve prompt clarity'). Uses pattern recognition across trace populations to identify common failure causes and recommend specific remediation actions. Insights are ranked by impact (percentage of failures they would address) and actionability.

Integrates with NVIDIA NeMo for dataset and metric customization, NVIDIA NIM for real-time observability of NIM-deployed systems, and NVIDIA Guardrails for safety/security enforcement via 'Galileo Protect'. Enables users to apply Galileo evaluation and guardrails to NVIDIA-deployed LLM systems without additional instrumentation.