Dagster

Dagster's core asset system uses Python decorators (@asset) to define data assets as first-class objects with explicit dependency graphs. Unlike traditional DAGs that model tasks, Dagster's asset-centric model tracks data lineage and materialization state directly. The system builds a directed acyclic graph of asset dependencies at definition time, enabling automatic scheduling, backfilling, and impact analysis across the entire data lineage.

Dagster implements a pluggable I/O manager system that handles serialization, deserialization, and storage of asset outputs with full type checking. Each asset can declare input/output types (Python type hints), and the framework validates data at materialization time. I/O managers are resource-based, allowing different storage backends (S3, Snowflake, local filesystem, etc.) to be swapped without changing asset definitions. The system supports both in-memory and persistent storage with automatic schema validation.

Dagster's asset health system tracks the freshness and status of assets based on materialization time and custom health checks. The system supports freshness policies (e.g., 'must be materialized daily') that are evaluated by the asset daemon, triggering re-materialization if assets become stale. Custom health checks can be defined as Python functions that assess asset quality (row counts, schema validation, etc.). Asset health status is persisted and queryable via GraphQL, enabling monitoring dashboards and alerting. The system integrates with dbt test results for test-based health tracking.

Dagster's execution engine supports launching multiple runs for different asset partitions in parallel, with automatic partition key mapping across dependencies. The backfill system enables selecting specific asset partitions and automatically generating run requests for all affected downstream assets. The system tracks backfill progress and supports cancellation/resumption. Execution can be distributed across multiple workers using executors (in-process, multiprocess, Kubernetes, Celery), with automatic work distribution and resource management.

Dagster+ is a managed cloud service offering that provides hosted Dagster instances with built-in infrastructure, monitoring, and team collaboration features. It includes managed code locations (serverless execution), automatic scaling, integrated monitoring dashboards, and RBAC for team access control. Dagster+ abstracts away infrastructure management (Kubernetes, databases, etc.), enabling teams to focus on pipeline development. The service supports multiple deployment options (single-tenant, multi-tenant) and integrates with cloud providers (AWS, GCP, Azure).

Dagster's metadata system enables attaching arbitrary key-value metadata to assets, runs, and events for governance and discovery. Assets can be tagged with custom tags (owner, domain, sensitivity level) that are queryable and filterable. Metadata can include descriptions, SLAs, data quality thresholds, and custom domain-specific information. The system supports metadata inference from external sources (dbt tags, database schemas) and enables metadata-driven automation (e.g., triggering different actions based on asset tags). Metadata is persisted and queryable via GraphQL.

Dagster's automation layer uses sensors (event-driven triggers) and schedules (time-based triggers) to declaratively define when assets should materialize. Sensors poll external systems (S3, databases, APIs) or listen to Dagster events, while schedules use cron expressions or custom tick functions. The asset daemon continuously evaluates sensor/schedule conditions and creates runs when triggered. Dynamic partitions allow sensors to create new partitions at runtime based on external data (e.g., new S3 prefixes), enabling adaptive pipelines that scale with data growth.

Dagster's partitioning system enables dividing assets into logical chunks (daily, hourly, by tenant, by region) with support for multi-dimensional partition spaces. Partition definitions are declarative objects (DailyPartitionsDefinition, StaticPartitionsDefinition, DynamicPartitionsDefinition) that define the partition key space. Assets can depend on specific partitions of upstream assets, and the system automatically maps partition keys through the dependency graph. Backfills operate at partition granularity, allowing selective re-materialization of historical data without full asset re-runs.

Dagster's dbt integration (via dagster-dbt library) automatically ingests dbt projects and materializes dbt models as Dagster assets with full lineage preservation. The system parses dbt manifests to extract model dependencies, tags, and metadata, creating asset definitions without manual code. Dagster can orchestrate dbt runs (dbt run, dbt test) as asset materializations, track dbt test results as asset health indicators, and integrate dbt lineage with non-dbt assets in the same graph. The integration supports both local dbt projects and dbt Cloud APIs.

Dagster exposes a comprehensive GraphQL API (dagster-graphql package) for querying execution history, asset metadata, and event logs. The API supports complex queries for run status, asset materialization events, sensor/schedule state, and partition status. Clients can subscribe to real-time event streams, trigger runs programmatically, and retrieve asset lineage. The GraphQL schema is auto-generated from Python type definitions, ensuring consistency between CLI/UI and API. The Dagster UI itself uses this API, making it the canonical interface for external integrations.

Dagster's execution model is built on structured events (DagsterEvent objects) that capture all execution details: asset materializations, step outputs, logs, errors, and custom events. Events are persisted to an event log store (configurable: SQLite, PostgreSQL, etc.) with full context including run ID, step key, and timestamp. The system supports custom event types via DagsterEventType, enabling domain-specific observability. Event logs are queryable via GraphQL and CLI, and can be streamed to external systems (Datadog, New Relic, etc.) via event handlers.

Dagster's resource system provides a declarative way to inject dependencies (database connections, API clients, credentials) into assets and ops. Resources are defined as classes or functions decorated with @resource, and are bound to assets via the context parameter. The system supports resource initialization/cleanup (setup/teardown), resource composition (resources depending on other resources), and environment-specific configuration. Resources are instantiated once per run and passed to all assets in that run, enabling efficient connection pooling and state sharing.

Dagster's workspace system organizes definitions (assets, jobs, schedules, sensors) into code locations that can be loaded dynamically. Code locations are Python modules or packages that export a Definitions object, and can be loaded from local filesystem, Python packages, or remote URLs. The workspace.yaml file specifies which code locations to load, enabling multi-team development where each team maintains their own definitions. The system supports dynamic code location discovery and hot-reloading without restarting the daemon, enabling rapid iteration.

Dagster's Pipes framework enables orchestrating external processes (shell scripts, Spark jobs, dbt runs, Python subprocesses) as first-class assets with full observability. Pipes uses a lightweight protocol to capture outputs and events from external processes, streaming them back to Dagster for logging and event tracking. The framework supports multiple execution contexts (local, Kubernetes, Databricks, Spark) with a unified API. External processes emit structured events via the Pipes protocol, enabling Dagster to track their progress and capture outputs without polling or log parsing.