great-expectations

Enables developers to write data quality tests as Python code using an Expectation-based DSL that encodes business logic and data contracts. Tests are expressed declaratively (e.g., 'column X must be non-null', 'values in column Y must be between 0-100') and compiled into executable validation rules that can be versioned, shared, and integrated into CI/CD pipelines. The framework abstracts away the complexity of implementing custom validation logic by providing a library of pre-built Expectation types covering common data quality patterns.

Provides a Checkpoint abstraction that bundles multiple Expectations and executes them at defined stages in a data pipeline (development, pre-downstream, production). Checkpoints can be triggered manually, on-schedule, or integrated into orchestration tools (Airflow, dbt, Prefect) to validate data at ingestion, transformation, and output stages. Results are collected and can trigger alerts, block downstream processing, or log to monitoring systems. The framework supports conditional validation logic and parameterized Expectations to adapt tests to different data contexts.

Great Expectations provides a framework for developing custom Expectations that extend the built-in library with domain-specific validation logic. Custom Expectations are implemented as Python classes that inherit from base Expectation classes and implement validation logic, rendering logic, and metadata. The framework handles execution, result collection, and integration with the standard validation pipeline. Custom Expectations can be packaged as plugins and shared across teams or published to the community. The framework supports custom Expectation validation, documentation generation, and testing utilities.

Provides an AI-assisted test generation feature (ExpectAI) that analyzes sample data and automatically generates Expectation Suites reflecting observed data patterns and statistical properties. The system infers constraints on column types, value ranges, null rates, and distributions, then suggests Expectations that encode these patterns. Generated tests can be reviewed, edited, and committed to version control. This reduces manual effort in bootstrapping data quality tests for new data sources or tables.

Executes Expectations and produces structured validation results (JSON/YAML) containing pass/fail status, failure counts, and diagnostic metadata for each Expectation. Results are aggregated into Validation Reports that can be rendered as HTML Data Docs—human-readable documentation showing data quality metrics, test results, and data lineage. Data Docs are versioned and can be hosted on static web servers or integrated into data catalogs. Results can also be exported to monitoring systems, data warehouses, or custom dashboards for real-time quality tracking.

Abstracts data source connectivity through a connector pattern, enabling Expectations to be executed against multiple data sources (SQL databases, Pandas DataFrames, Spark, Snowflake, BigQuery, Redshift, etc.) without changing test code. Connectors handle data fetching, query translation, and result collection. The framework supports both batch validation (full table scans) and sampling-based validation for large datasets. Connectors are extensible; custom connectors can be implemented for proprietary data systems.

GX Cloud provides a fully-managed SaaS platform that eliminates the need to self-host and manage Great Expectations infrastructure. The platform includes a web-based UI for test authoring, a managed validation execution engine, result storage, and Data Docs hosting. Teams can set up validation in minutes without deploying Python code or managing databases. GX Cloud includes features like ExpectAI, real-time monitoring dashboards, team collaboration tools, and integrations with data orchestration platforms. Pricing tiers (Developer free, Team, Enterprise) support different team sizes and feature sets.

Expectation Suites are stored as JSON/YAML configuration files that can be versioned in Git, enabling data quality tests to be treated as code. Suites are decoupled from specific data sources, allowing the same suite to be executed against different tables or databases without modification. Configuration management supports parameterization (e.g., table name, column names, thresholds) enabling test reuse across similar datasets. Suites can be organized hierarchically and shared across teams. The framework supports suite validation, merging, and conflict resolution for collaborative workflows.

Provides native or community-supported integrations with popular data orchestration tools (Airflow, dbt, Prefect, Dagster) and CI/CD systems (GitHub Actions, GitLab CI, Jenkins). Integrations enable Checkpoints to be triggered as pipeline steps, with results blocking downstream tasks on failure or logging to pipeline metadata. GX provides Airflow operators, dbt test adapters, and webhook-based triggers for other platforms. Results can be exported to orchestration logs, monitoring systems, or custom notification channels. Integration patterns support both synchronous (blocking) and asynchronous (non-blocking) validation modes.

GX Cloud provides real-time monitoring dashboards that track validation results, data quality metrics, and trends over time. Dashboards display pass/fail rates, failure counts, and historical patterns for each Expectation and Checkpoint. Alerting rules can be configured to trigger notifications (email, Slack, webhooks) when quality thresholds are breached or validation failures occur. Alerts support conditional logic (e.g., alert only if failure rate exceeds 10%) and can be routed to different teams based on data ownership. Monitoring data is retained for historical analysis and trend detection.

GX Cloud provides team collaboration features including shared Expectation Suites, collaborative test authoring, and role-based access control (RBAC). Teams can assign roles (Admin, Editor, Viewer) to control who can create, edit, or view Expectations and validation results. Audit logs track changes to Expectations and validation configurations. Workspace organization enables teams to manage multiple data sources and pipelines within a single GX Cloud account. Notifications and mentions enable team communication around data quality issues.