Elementary

Elementary generates dbt test macros that collect time-series metrics (row counts, freshness, schema changes) directly within dbt runs and apply statistical anomaly detection algorithms (z-score, IQR, moving average baselines) to flag deviations. Tests execute natively in dbt's DAG, storing results in Elementary's metadata schema, eliminating separate monitoring infrastructure and enabling anomalies to fail dbt runs.

Elementary's dbt package and CLI parse dbt artifacts (manifest.json, run_results.json) to extract test metadata, execution times, and failure reasons, then correlates test failures with downstream model dependencies to surface which datasets are affected. Stores test lineage in Elementary's metadata schema, enabling root-cause analysis by tracing failures upstream through the DAG.

Elementary Cloud provides a managed SaaS platform that syncs monitoring data from open-source Elementary instances, enabling team collaboration, centralized dashboards, and advanced features (column-level lineage, AI-powered tests, team management). Cloud instances pull data from warehouse via Elementary CLI's `send-report` command or push via API, maintaining data residency while providing collaborative UI.

Elementary CLI collects anonymous telemetry (command usage, feature adoption, error rates) via optional tracking module (elementary/tracking/tracking_interface.py) to inform product development. Tracking is opt-out and does not collect sensitive data (SQL, credentials, table names), enabling Elementary team to understand adoption patterns without compromising user privacy.

Elementary enables teams to define monitoring configuration (anomaly detection thresholds, freshness SLAs, alert routing) directly in dbt YAML files using the 'meta' field on models and columns. This approach treats monitoring configuration as code, enabling version control, code review, and reproducible monitoring setups. Configuration includes owner tags (meta.owner), anomaly detection parameters (meta.anomaly_detection), and custom metric definitions. The dbt package reads this configuration during runs to apply monitoring logic without separate configuration files.

Elementary CLI's `report` command generates a self-contained HTML dashboard aggregating test results, anomaly detections, model performance metrics, and data lineage into a single interactive report. The `send-report` command distributes reports via Slack, Teams, email, or uploads to S3/GCS, enabling async sharing of data quality status without requiring dashboard access.

Elementary's warehouse client layer abstracts SQL dialects across Snowflake, BigQuery, Redshift, Databricks, and Postgres, providing a unified interface for querying metadata (table schemas, row counts, freshness timestamps, column statistics). Clients handle dialect-specific syntax for information_schema queries, enabling anomaly detection and lineage analysis to work identically across warehouses without custom logic per platform.

Elementary's alerting system processes test failures and anomalies through a configuration-driven pipeline that filters alerts by severity/tags, groups related failures (e.g., all failures in a data mart), and routes to different channels (Slack, Teams, email) based on owner tags or custom rules. Alert deduplication prevents duplicate notifications for the same failure across multiple runs.

Elementary CLI implements dbt's selector syntax (--select, --exclude, --selector) to filter which models and tests to monitor, enabling targeted monitoring of specific data marts or critical paths. Selector evaluation happens during CLI execution, allowing users to run `edr monitor --select tag:critical` to monitor only critical models without modifying dbt configuration.

Elementary's schema monitoring tests detect additions, deletions, and type changes in table columns by comparing current schema against historical snapshots stored in Elementary's metadata tables. Tests execute natively in dbt, flagging schema changes as test failures and enabling alerts when unexpected schema modifications occur (e.g., column drops in production).

Elementary tracks dbt model execution metrics (runtime, row counts, resource utilization) across runs and compares against configurable SLAs (e.g., 'model must complete in <5 minutes'). Performance degradation is detected via statistical analysis of historical runtimes, enabling alerts when models exceed expected execution time or resource consumption.

Elementary monitors table freshness by tracking the timestamp of the last data update (via dbt's `updated_at` metadata or warehouse-specific last_modified timestamps). Freshness tests compare current time against the last update timestamp and alert when data exceeds configured freshness SLAs (e.g., 'data must be updated within 24 hours').

Elementary CLI parses dbt artifacts (manifest.json, run_results.json) generated by both dbt Cloud and dbt Core, extracting test results, model metadata, and execution logs. Supports both local artifact paths and remote dbt Cloud API integration, enabling Elementary to work with any dbt deployment model without requiring dbt-specific SDKs.