Elementary

Elementary generates dbt test macros that collect time-series metrics (row counts, column distributions, freshness) and apply statistical anomaly detection algorithms (z-score, moving average, seasonal decomposition) directly within the dbt DAG. Tests execute during dbt run/test phases, storing metric history in a metadata schema for trend analysis. This approach embeds observability into dbt's native execution model rather than post-processing logs, enabling anomalies to be detected and surfaced as test failures within standard dbt workflows.

Elementary monitors dbt model schemas by comparing column definitions, types, and constraints across runs using dbt artifacts (manifest.json, run_results.json). It tracks schema changes (added/removed/modified columns) and builds end-to-end data lineage by parsing dbt model dependencies and test relationships. The system stores lineage metadata in a warehouse schema and correlates test failures with upstream model changes to identify root causes. Column-level lineage (available in Cloud) traces data flow through transformations to pinpoint which upstream columns affect downstream failures.

Elementary supports uploading generated reports to AWS S3 or Google Cloud Storage (GCS) for centralized archival and sharing. The system stores report URLs and metadata in warehouse tables for historical tracking. Reports can be accessed via direct URLs or embedded in dashboards. Cloud storage integration requires credential configuration (AWS access keys or GCS service account) and supports configurable bucket paths and retention policies.

Elementary Cloud is a managed SaaS platform that extends the open-source CLI with team collaboration features, column-level lineage tracking, AI-powered test generation, and centralized dashboard. The Cloud platform stores monitoring data in Elementary's managed infrastructure, eliminating the need for teams to manage warehouse metadata tables. It provides role-based access control (RBAC), team management, and advanced features like automated test recommendations and data catalog exploration. Cloud setup involves connecting dbt Cloud projects and configuring data warehouse credentials through the web UI.

Elementary enables teams to define monitoring configuration (anomaly detection thresholds, freshness SLAs, alert routing) directly in dbt YAML files using the 'meta' field on models and columns. This approach treats monitoring configuration as code, enabling version control, code review, and reproducible monitoring setups. Configuration includes owner tags (meta.owner), anomaly detection parameters (meta.anomaly_detection), and custom metric definitions. The dbt package reads this configuration during runs to apply monitoring logic without separate configuration files.

Elementary monitors data freshness by tracking the timestamp of the most recent data update in each model (via dbt-generated updated_at columns or custom timestamp columns). It compares the latest data timestamp against the current time to calculate staleness and generates alerts when data exceeds configured freshness thresholds (e.g., 'data must be updated within 24 hours'). Freshness checks execute as dbt tests that query the warehouse to measure time-since-last-update, enabling freshness monitoring without external schedulers.

Elementary CLI parses dbt test execution results (from run_results.json and warehouse test tables) to aggregate pass/fail status, execution time, and failure messages across all dbt tests. It correlates test failures with model changes, data anomalies, and schema modifications to provide root cause analysis. The system groups related test failures and generates summaries highlighting which tests failed, which models are affected, and what changed upstream. Test metadata is stored in warehouse tables for historical analysis and trend tracking.

Elementary generates interactive HTML data quality reports that visualize test results, anomalies, freshness metrics, and model performance over time. The report builder queries warehouse metadata tables to construct dashboards showing test pass rates, anomaly trends, and data lineage. Reports can be distributed via Slack, Teams, email, or uploaded to cloud storage (S3, GCS) for sharing with stakeholders. The CLI command 'edr report' generates reports locally, and 'edr send-report' uploads them to cloud storage or messaging platforms with configurable scheduling.

Elementary processes generated alerts (from test failures, anomalies, freshness violations) through a filtering and grouping engine that deduplicates related alerts, groups them by model/owner, and applies tag-based routing rules. Alerts can be tagged with owner information from dbt model YAML (meta.owner field) to route notifications to responsible teams. The system supports alert suppression rules (e.g., 'ignore freshness alerts on paused models') and severity-based filtering to reduce alert fatigue. Alert metadata is stored in warehouse tables for audit and trend analysis.

Elementary abstracts warehouse-specific SQL dialects and APIs through a normalized metadata collection layer that works across Snowflake, BigQuery, Redshift, Databricks, PostgreSQL, and DuckDB. The system executes warehouse-agnostic SQL queries (using dbt's adapter abstraction) to extract test results, model metadata, and performance metrics, then normalizes the results into a common schema stored in Elementary's metadata tables. This approach enables the same monitoring logic to run on any supported warehouse without code changes.

Elementary provides a dbt package (installed via dbt_packages) containing reusable macros that execute during dbt runs to collect metrics (row counts, column distributions, freshness timestamps) and store them in warehouse metadata tables. The package includes pre-built macros for common metrics (volume, freshness, schema) and allows custom metric definitions via dbt YAML configuration. Metrics are collected incrementally during each dbt run and appended to time-series tables, enabling historical trend analysis without external metric collection infrastructure.

Elementary provides a Python CLI (edr) that orchestrates monitoring workflows by executing dbt commands, querying warehouse metadata, and triggering alerts/reports. The CLI includes commands for 'monitor' (detect anomalies and send alerts), 'report' (generate HTML reports), 'send-report' (distribute reports), and 'debug' (troubleshoot setup). The CLI manages configuration from YAML files, handles warehouse connections via dbt profiles, and coordinates multi-step workflows (run dbt → collect metrics → detect anomalies → send alerts) without requiring external orchestrators.

Elementary integrates with Slack and Microsoft Teams via webhook URLs to send alert notifications with customizable message formatting. Alerts include test failure summaries, anomaly details, affected models, and recommended actions. The system supports channel-based routing (different alerts to different channels based on model owner tags) and message threading to group related alerts. Webhook configuration is stored in Elementary's config files, enabling teams to route alerts without code changes.