Meltano

Meltano enables users to define complete Extract-Load-Transform pipelines declaratively in meltano.yml, which specifies extractors (Singer taps), loaders (Singer targets), transformers (dbt), and inline mappers as sequential execution blocks. The configuration system uses multi-layer settings resolution (environment variables, project config, plugin defaults) to manage plugin parameters without code changes. The CLI parses this YAML and orchestrates plugin execution through isolated virtual environments managed per plugin.

Meltano abstracts the Singer protocol (JSON-based streaming format for data integration) through a plugin system that discovers, installs, and invokes 600+ pre-built Singer taps (extractors) and targets (loaders) from Meltano Hub. Each plugin runs in an isolated virtual environment (managed via uv or virtualenv) with its own dependencies, and Meltano handles stdin/stdout piping between tap and target processes, managing state files for incremental replication. The Singer protocol integration layer translates plugin configurations into command-line invocations and parses Singer messages (SCHEMA, RECORD, STATE) for state persistence.

Meltano implements a Logging System that captures detailed execution logs from all pipeline components (extractors, loaders, transformers, mappers) and stores them in a centralized log directory. The system supports multiple log levels (DEBUG, INFO, WARNING, ERROR) and can output logs to console and file simultaneously. Meltano also provides a Telemetry and Analytics system that collects anonymous usage data (command execution, plugin usage, error rates) to improve the platform. Users can disable telemetry via configuration, and all telemetry data is anonymized and sent to Meltano's analytics backend.

Meltano's Plugin Configuration and Inheritance system allows plugins to inherit configuration from parent definitions and environment-specific overrides, enabling DRY (Don't Repeat Yourself) configuration patterns. Users can define base plugin configurations in meltano.yml and override specific settings per environment (dev/staging/prod) or per pipeline variant. The system supports configuration inheritance chains where plugins inherit from base definitions, and environment variables can override any inherited setting. This enables a single plugin definition to serve multiple use cases without duplication.

Meltano generates and maintains a meltano.lock file that pins exact versions of all installed plugins, enabling reproducible installations across team members and CI/CD environments. The lock file is generated during meltano install and tracks plugin versions, variant selections, and dependency hashes. Users can commit meltano.lock to version control to ensure all team members use identical plugin versions. The system supports lock file updates via meltano update command, and users can manually edit lock files for version overrides or dependency resolution.

Meltano provides persistent state management for incremental data replication, storing Singer protocol STATE messages in configurable backends (local filesystem, S3, GCS, Azure Blob Storage). The state system tracks bookmarks (e.g., last-modified timestamp, cursor position) per tap-target pair, enabling subsequent runs to fetch only new/changed records. State is retrieved before pipeline execution and persisted after successful completion, with support for state reset and manual state editing via CLI commands. The architecture decouples state storage from execution, allowing state to be shared across distributed pipeline runs.

Meltano provides a CLI-driven plugin discovery and installation system that queries Meltano Hub (600+ pre-built Singer taps/targets) and installs plugins into isolated Python virtual environments using uv or virtualenv. The meltano add command discovers plugins by name, resolves dependencies, and creates a plugin lock file (meltano.lock) tracking installed versions. Each plugin gets its own virtual environment to prevent dependency conflicts, and Meltano manages environment activation during pipeline execution. The plugin system supports custom plugins (local Python packages or git repositories) alongside Hub plugins.

Meltano implements a hierarchical settings resolution system that merges configuration from multiple sources: environment variables, meltano.yml project file, plugin defaults, and system settings. The Settings Service Architecture resolves plugin parameters by checking sources in priority order (environment variables override project config, which overrides plugin defaults), enabling environment-specific configurations without duplicating pipeline definitions. Configuration supports variable interpolation (e.g., ${MELTANO_ENVIRONMENT}) and environment-specific overrides (dev/staging/prod). The system also handles sensitive values (passwords, API keys) by supporting environment variable references.

Meltano integrates dbt (data build tool) as a native transformer plugin within ELT pipelines, allowing users to compose dbt runs as pipeline blocks alongside extractors and loaders. The integration invokes dbt commands (dbt run, dbt test, dbt snapshot) as part of the pipeline execution flow, with Meltano managing dbt project discovery, manifest parsing, and state passing between pipeline steps. Users define dbt transformations in meltano.yml and execute them via meltano run commands, enabling end-to-end ELT orchestration without separate dbt orchestration tools.

Meltano provides a CLI-based pipeline execution system where users compose pipelines as sequences of 'blocks' (extractors, loaders, transformers, mappers) via the meltano run command. The Block System parses the pipeline definition, validates plugin availability, and executes blocks sequentially with stdin/stdout piping between compatible blocks (e.g., tap output piped to target input). The Runners subsystem handles process management, environment activation, and error handling. Users can define reusable pipeline templates in meltano.yml and invoke them with different parameters, enabling parameterized pipeline execution.

Meltano provides a scheduling system (Scheduling and Task Sets) that allows users to define scheduled jobs in meltano.yml, specifying pipeline names, execution intervals (cron expressions), and job metadata. The meltano schedule command manages job definitions (create, list, enable/disable), and Meltano can integrate with system schedulers (cron, Windows Task Scheduler) or external orchestrators (Airflow, Prefect) via CLI invocation. Task sets group related pipelines for batch execution, enabling users to run multiple pipelines in sequence with a single command.

Meltano supports inline mappers (data transformation plugins) that process records between extraction and loading, enabling lightweight transformations without requiring a separate transformation layer. Mappers receive Singer RECORD messages from the tap, apply transformations (field selection, renaming, type casting, filtering), and emit transformed RECORD messages to the target. The mapper system integrates with the Singer protocol pipeline, allowing mappers to be composed as blocks in meltano run commands. Meltano provides built-in mappers (e.g., meltano-map-transformer) and supports custom mappers via the plugin system.

Meltano provides a meltano init command that scaffolds a new Meltano project with a default directory structure, meltano.yml configuration file, and optional sample plugins. The Project System manages project discovery (finding meltano.yml in the current or parent directories) and activation (setting up the project environment). The initialization process creates a .meltano directory for internal state, plugin virtual environments, and logs. Users can initialize projects with optional templates or examples to accelerate setup.