Great Expectations

Enables data teams to define data quality rules declaratively using a fluent Python API that chains expectation methods (e.g., expect_column_values_to_be_in_set, expect_table_row_count_to_be_between). Expectations are serialized as JSON and stored in ExpectationSuite objects, allowing version control and reuse across validation runs. The system supports 50+ built-in expectation types covering schema, distribution, and custom metrics.

Executes expectations against data using pluggable execution engines (Pandas, SQL, Spark, Databricks) by translating expectation definitions into engine-specific queries through a Metric Provider system. Each expectation maps to metrics (e.g., column_values, table_row_count) that are computed differently per engine — SQL expectations compile to WHERE clauses, Pandas uses vectorized operations, Spark uses DataFrame API. The Validator class orchestrates metric computation and result aggregation.

Provides cloud-hosted validation management through GX Cloud, which centralizes expectations, validation runs, and data quality insights across teams. GX Cloud agents run validation checkpoints on schedule and report results to the cloud backend, enabling web-based dashboards, team collaboration, and audit trails. The cloud platform supports role-based access control, validation scheduling, and integration with data sources (Snowflake, Redshift, Databricks) without requiring local infrastructure.

Enables teams to define custom metrics by subclassing MetricProvider and implementing compute methods for each execution engine (Pandas, SQL, Spark). Custom metrics are registered with the MetricProvider registry and can be used in expectations without modifying core GX code. The system supports metric parameters (e.g., 'column_name', 'threshold') and caching of metric results to avoid redundant computation.

Generates ExpectationSuites automatically by analyzing data distributions using the Rule-Based Profiler, which applies heuristic rules to infer expectations (e.g., 'if a column has <10 unique values, expect values to be in set'). The profiler computes statistical metrics (cardinality, nullness, data types, value ranges) and applies configurable rules to suggest expectations. Results are stored as ExpectationSuites that can be reviewed, edited, and deployed without manual definition.

Organizes validation runs into Checkpoints, which bundle a set of ExpectationSuites, data assets, and validation actions (e.g., send alert, update metadata) into a single executable unit. Checkpoints can be scheduled via Airflow, Prefect, or cron, and support conditional actions based on validation results (e.g., 'if validation fails, trigger PagerDuty alert'). The Checkpoint system stores validation history and provides a unified interface for monitoring data quality across pipelines.

Provides a DataContext abstraction that manages configuration, expectations, validation results, and metadata through pluggable store backends (FileSystemStore, S3Store, DatabaseStore, GCSStore). The context system supports both file-based (YAML config) and cloud-based (GX Cloud) deployments, with stores handling persistence of expectations, validation results, and data docs. Stores are backend-agnostic, allowing teams to swap storage without changing application code.

Generates HTML documentation of expectations, validation results, and data quality metrics using a Site Builder that composes Page Renderers for different content types (ExpectationSuite pages, validation result pages, data asset pages). Renderers transform ExpectationSuite and ValidationResult objects into HTML using Jinja2 templates, with support for custom CSS and JavaScript. Data Docs are published to FileSystem, S3, or GCS and can be embedded in data catalogs or served as standalone sites.

Executes actions (email, Slack, webhook, update metadata) based on validation outcomes through a pluggable ValidationAction system. Actions are triggered after checkpoint validation completes and receive ValidationResult objects, enabling conditional logic (e.g., 'send alert only if validation failed'). Built-in actions include EmailAction, SlackNotificationAction, UpdateDataDocsAction, and custom actions can be implemented by subclassing ValidationAction.

Organizes data into Batches, which represent immutable snapshots of data assets at specific points in time, enabling validation of specific data versions and tracking of data lineage. Batches are identified by batch_id (e.g., 'daily_2024-01-15') and store metadata (creation time, data source, asset name) in the metadata store. The Batch system integrates with DataSources to enable automatic batch discovery and supports manual batch creation for ad-hoc validation.

Provides a fluent Python API for configuring data sources dynamically without YAML, enabling programmatic creation of SQL datasources, Pandas datasources, and Spark datasources with batch discovery rules. The API supports method chaining (e.g., datasource.add_table_asset(...).add_batch_definition(...)) and generates batch identifiers automatically based on partition keys or file paths. Datasources are stored in the DataContext and can be referenced by name in expectations and checkpoints.

Persists validation results (pass/fail status, metrics, exception details) to a metadata store (FileSystem, S3, database) and provides query APIs to retrieve results by batch, expectation, or time range. ValidationResult objects are serialized to JSON and indexed by batch_id, expectation_suite_name, and run_id, enabling efficient retrieval of validation history. The metadata store supports filtering and aggregation queries for trend analysis and SLO monitoring.