Hamilton

Transforms decorated Python functions into nodes within a directed acyclic graph by parsing function signatures and dependency annotations. Hamilton introspects function parameters to automatically infer data flow edges, building a complete lineage graph without explicit edge declarations. This enables automatic tracking of which transformations depend on which inputs, supporting end-to-end data provenance from raw inputs to final outputs.

Executes compiled DAGs across multiple execution backends (local, Dask, Pandas, Spark, Ray) through a unified driver abstraction layer. Hamilton decouples the DAG definition from execution strategy, allowing the same pipeline code to run locally for development, on Dask for distributed processing, or on Spark for production without code changes. Drivers handle resource allocation, parallelization, and result collection.

Provides built-in connectors and patterns for reading from and writing to external systems (databases, data lakes, APIs, message queues). Hamilton includes @extract nodes for data ingestion and patterns for writing results to external systems, abstracting away connection management and format conversion. Connectors handle authentication, connection pooling, and error handling.

Tracks execution metrics (timing, memory, task status) and provides APIs to inspect pipeline performance. Hamilton logs execution time per node, memory consumption, and task status, enabling identification of bottlenecks and performance regressions. Metrics can be exported to monitoring systems (Prometheus, CloudWatch) or analyzed locally for optimization.

Enables runtime parameterization of DAG execution through a configuration system that overrides function inputs without modifying source code. Hamilton accepts configuration dictionaries or YAML files that map parameter names to values, allowing the same DAG to execute with different inputs (e.g., different data sources, thresholds, or feature sets) by changing config rather than code. Parameters propagate through the DAG automatically.

Provides APIs to execute individual nodes or subgraphs of the DAG interactively, returning intermediate results for inspection. Hamilton allows developers to execute a single transformation node or a chain of nodes without running the entire pipeline, enabling exploratory data analysis and debugging. Results are returned as native Python objects (DataFrames, dicts, etc.) for immediate inspection in notebooks or REPL environments.

Generates test scaffolding and enables unit testing of individual transformation nodes in isolation. Hamilton introspects node signatures and generates test templates that mock dependencies, allowing developers to test a single function without executing upstream nodes. Tests can verify output types, value ranges, or specific transformations without requiring full pipeline execution or external data.

Generates visual representations of the compiled DAG as directed graphs, showing nodes (transformations) and edges (data dependencies). Hamilton exports DAGs to multiple formats (Graphviz, Mermaid, HTML) for visualization in notebooks, documentation, or external tools. The visualization includes node metadata (input/output types, execution time) and can highlight critical paths or problematic nodes.

Validates node inputs and outputs against declared types and optional schema constraints. Hamilton uses Python type hints to enforce data types at node boundaries, catching type mismatches before execution. Optional schema validation (via Pydantic or custom validators) can enforce constraints like column presence, value ranges, or data distributions, enabling early detection of data quality issues.

Organizes transformations into reusable modules using Python packages and namespace conventions. Hamilton allows developers to define transformation functions across multiple files and modules, automatically discovering and composing them into a single DAG. Namespacing prevents naming conflicts and enables selective node inclusion, allowing teams to build large pipelines from composable, independently-testable modules.

Caches node outputs and skips re-execution of unchanged nodes in subsequent pipeline runs. Hamilton tracks node inputs and code, detecting when a node's dependencies or implementation have not changed, and reuses cached results instead of re-computing. This enables fast iteration during development and reduces redundant computation in production pipelines, particularly for expensive transformations.

Provides a decorator-based plugin system allowing developers to define custom node types with specialized behavior. Beyond @node and @extract, Hamilton supports custom decorators that can modify execution (e.g., @retry, @cache, @validate), add metadata, or integrate with external systems. The decorator system is composable, allowing multiple decorators to be stacked on a single function.