For data engineering, the strongest fits are Apache Airflow for programmatically authoring, scheduling and monitoring workflows 3, Dagster as a data orchestrator for building, scheduling and monitoring reliable pipelines 1, and dbt for building modular, tested, version-controlled SQL transformations 2. Databricks Lakeflow and Snowflake cover pipeline development and loading within their platforms 510.
- 1
Apache Airflowapache.orgverified and signed
Platform created by the community to programmatically author, schedule and monitor workflows in standard Python 3.
official:API · Airflow API (Stable)API · Airflow API (Stable)+22 community tools built for it
for agents: pipelines · airflow · integration
- 2
- 3
dbtgetdbt.comverified and signed
Builds data pipelines as modular, tested, version-controlled SQL models with lineage across projects and columns 2.
for agents: using-dbt-for-analytics-engineering · migrating-dbt-project-across-platforms · adapters
- 4
Databricks Lakeflow Spark Declarative Pipelinesdatabricks.comverified and signed
Develops Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables) for batch or streaming data pipelines on Databricks 15.
official:SDK · databricks-cliskill · databricks-coreskill · databricks-appsskill · databricks-pipelinesskill · databricks-jobs+77 community tools built for it
for agents: databricks-pipelines · databricks · databricks-jobs
- 5
Snowflake AI Data Cloudsnowflake.comverified and signed
Fully managed platform for ingesting, processing and modeling data, including loading data into tables from stages with COPY INTO 1020.
official:SDK · snowflake-connector-pythonSDK · snowflake-sqlalchemy+64 community tools built for it
for agents: analytics · load_data_from_stage · analytics
Sources
- 1Modern Data Orchestrator Platform | Dagsterdagster.io · read 2026-10-29
- 2Deliver trusted data with dbt | dbt Labsgetdbt.com · read 2026-10-29
- 3Apache Airflowairflow.apache.org · read 2026-10-29
- 5Databricks documentation | Databricksdatabricks.com · read 2026-10-29
- 10Snowflake AI Data Cloudsnowflake.com · read 2026-10-29
- 15databricks-pipelines: declares "databricks-pipelines" (the record we read and signed)unfragile.ai · read 2026-10-24
- 20snowflake-sqlalchemy: declares "analytics" (the record we read and signed)unfragile.ai · read 2026-10-24
For agents: the 10 actions behind this answer
· Dagster website read, tools not yet checked
Dagster is the data orchestrator platform that helps you build, schedule, and monitor reliable data pipelines—fast, flexible, and built for teams.
· Getdbt Metricflow website read, tools not yet checked
dbt Labs empowers data teams to build reliable, governed data pipelines—accelerating analytics and AI initiatives with speed and confidence.
pipelines · apache-airflow verified and signed
bigquery-pipeline-audit · bigquery-pipeline-audit verified and signed
Audits Python + BigQuery pipelines for cost safety, idempotency, and production readiness. Returns a structured report with exact patch locations.
databricks-pipelines · databricks-pipelines verified and signed
Develop Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables) on Databricks. Use when building batch or streaming data pipelines with Python or SQL, including Auto CDC from event streams or periodic complete snapshots. Invoke BEFORE starting implementation.
using-dbt-for-analytics-engineering · using-dbt-for-analytics-engineering verified and signed
Builds and modifies dbt models, writes SQL transformations using ref() and source(), creates tests, and validates results with dbt show. Use when doing any dbt work - building or modifying models, debugging errors, exploring unfamiliar data sources, writing tests, or evaluating impact of changes.
· Prefect website read, tools not yet checked
Orchestrate workflows with Prefect. Connect AI agents to your tools and data with FastMCP Gateway. Open-source foundations, production-ready platforms.
· Datanika Your Data Pipeline Platform website read, tools not yet checked
Build, orchestrate, and monitor data pipelines with dlt and dbt. Multi-tenant, multi-language, open-source.
databricks · mlflow verified and signed
· Langchain Google website read, tools not yet checked
LangChain enables every company to own their intelligence. Control, govern, and compound intelligence with an open agent engineering platform.
Labels are the instruments' verdict and are never changed by ranking. Agents and robots get this same answer as JSON at /api/v1/find?q=…&answer=1 and through the MCP server.