airflow

Airflow represents workflows as Directed Acyclic Graphs (DAGs) where tasks are nodes and dependencies are edges. The scheduler parses Python DAG definitions, builds the dependency graph at runtime, and executes tasks in topologically-sorted order with support for conditional branching, dynamic task generation, and cross-DAG dependencies. This approach enables declarative workflow definition in code rather than configuration files, allowing programmatic task generation and complex dependency patterns.

Airflow decouples task scheduling from execution through an executor abstraction layer supporting multiple backends: SequentialExecutor (single-process), LocalExecutor (multiprocessing), CeleryExecutor (distributed message queue), KubernetesExecutor (containerized tasks), and custom executors. Tasks are serialized, pushed to a message broker or queue, and executed by worker processes that pull and execute them, with results persisted back to the metadata database. This architecture enables horizontal scaling and heterogeneous task execution environments.

Airflow's scheduler is a long-running process that periodically parses DAGs, creates task instances for scheduled execution dates, and submits them to executors. Scheduling is defined via schedule_interval (cron expression or timedelta) on each DAG. The scheduler maintains a heartbeat loop that checks for DAGs to schedule, monitors task progress, and enforces SLAs. Scheduling is time-based (not event-based), with configurable minimum scheduling interval (default 1 minute). The scheduler is single-threaded in early versions, becoming a bottleneck for large deployments.

Airflow provides Variables for storing configuration values (strings, JSON) in the metadata database, accessible to tasks via the Variable API. DAG and task parameters support Jinja2 templating, enabling dynamic value substitution at task execution time. Template variables include execution_date, run_id, task_id, and custom variables. This enables parameterized DAGs that adapt to execution context without code changes, supporting multi-environment deployments and dynamic configuration.

Airflow implements a pluggable logging system where task logs are written to local files by default but can be stored in remote backends (S3, GCS, Azure Blob Storage) via custom log handlers. Logs are streamed to the web UI from the configured log backend. The logging system captures task stdout/stderr, Airflow framework logs, and custom application logs. Log retention is configurable; old logs can be automatically deleted. This enables centralized log management and audit trails without requiring external logging infrastructure.

Airflow provides Sensor operators that poll external systems (S3, databases, HTTP endpoints, file systems) at configurable intervals until a condition is met, then trigger downstream tasks. Sensors implement exponential backoff, timeout handling, and poke modes (synchronous polling vs asynchronous deferral). This enables event-driven workflows where task execution depends on external state changes without requiring external event systems, though it trades efficiency for simplicity.

Airflow provides configurable retry logic at task level with exponential backoff, jitter, and max retry counts. Failed tasks can trigger alert callbacks, email notifications, or custom handlers. SLA (Service Level Agreement) monitoring tracks task execution time and triggers alerts if tasks exceed defined thresholds. Retry logic is implemented in the task execution loop, allowing tasks to be re-queued with exponential delay between attempts, while SLA checks run asynchronously in the scheduler.

Airflow provides XCom (cross-communication) as a key-value store for passing data between tasks. Tasks push values to XCom (serialized to JSON or pickle), and downstream tasks pull values by task_id and key. XCom is backed by the metadata database, enabling data persistence across task executions and worker processes. This decouples task execution from direct inter-process communication, but introduces serialization overhead and database I/O for every data exchange.

Airflow provides an Operator base class that encapsulates task logic and execution. Built-in operators handle common patterns: BashOperator (shell commands), PythonOperator (Python functions), SQLOperator (database queries), S3Operator (S3 operations), EmailOperator (email sending), and 100+ community operators for external systems (Spark, Kubernetes, Salesforce, etc.). Operators define task behavior, retry logic, and resource requirements declaratively, abstracting away execution details and enabling reusable task templates.

Airflow's scheduler supports backfilling — reprocessing historical data by running DAGs for past execution dates. The backfill command generates task instances for a date range, respecting task dependencies and retry logic. This enables reprocessing data after bug fixes, schema changes, or missed runs. Backfill is implemented as a separate execution path that creates task instances for historical dates and executes them through the normal scheduler/executor pipeline.

Airflow provides a Flask-based web UI displaying DAG structure, task status, execution history, logs, and metrics. The UI enables manual task triggering, task instance clearing (for re-execution), and DAG pause/unpause. Task logs are streamed from worker processes or log storage backends (S3, GCS). The UI is read-heavy but supports write operations (trigger, clear, pause) for operational control. This enables non-technical stakeholders to monitor pipelines and operators to debug failures without CLI access.

Airflow provides a Connections abstraction for storing credentials (database passwords, API keys, SSH keys) encrypted in the metadata database. Connections are referenced by name in tasks/operators, enabling credential rotation without DAG code changes. The Connections API supports multiple connection types (PostgreSQL, MySQL, S3, HTTP, SSH, etc.) with type-specific fields (host, port, login, password, extra JSON). Credentials are encrypted at rest using Fernet symmetric encryption with a configurable key.

Airflow supports multiple authentication backends (LDAP, Kerberos, OAuth, database) and role-based access control (RBAC) for the web UI. Authentication is pluggable via the auth_backend configuration; RBAC assigns roles (Admin, User, Viewer, Op) with granular permissions on DAGs, tasks, and UI features. Authorization is enforced at the web UI level, not at the task execution level. This enables multi-tenant deployments and compliance with organizational access policies.