Pipedream

Pipedream provides a component-based event source system where developers can trigger workflows from external APIs and services without writing connector code. Event sources are pre-built components (located in components/{app_name}/sources/) that poll or listen to webhooks from integrated services, emit structured events, and pass them to downstream workflow steps. The architecture uses a registry pattern where each source defines its own authentication, polling interval, and event schema, enabling instant activation of 1000+ integrations including Slack, GitHub, Stripe, Notion, and custom webhooks.

Pipedream allows developers to write custom code steps in Node.js, Python, Go, or Bash that execute within a workflow, with automatic access to previous step outputs, environment variables, and the Pipedream SDK. Each code step runs in an isolated serverless container with pre-installed common libraries and npm/pip package support, enabling developers to transform data, call external APIs, or implement custom logic without managing infrastructure. The execution environment provides a `steps` object that contains outputs from all previous workflow steps, enabling data flow between heterogeneous code steps.

Pipedream allows developers to define environment variables and secrets at the workflow or account level, which are injected into code steps and component configurations at runtime. Secrets are encrypted at rest and masked in logs to prevent accidental exposure. Environment variables can be referenced in YAML workflow definitions and accessed in code steps via `process.env`. The platform supports variable scoping (workflow-level vs account-level) and rotation policies.

Pipedream's ComponentForm system (@pipedream/connect-react) automatically generates interactive forms from component property schemas, with support for custom styling, theming, and validation. The form system renders different input types (text, select, checkbox, etc.) based on property definitions, handles dynamic property resolution, and provides real-time validation feedback. Developers can customize form appearance through CSS variables and theme configuration, or embed the form in external applications.

Pipedream provides a CI/CD pipeline for publishing components to the public registry, including automated validation, testing, and version management. Components are published from GitHub repositories using Pipedream's GitHub Actions, with automatic version bumping and changelog generation. The platform validates component code, schema definitions, and documentation before publishing. Published components are indexed in the Pipedream registry and available to all users.

Pipedream enables developers to transform and map data between workflow steps using the `steps` object, which provides type-safe access to outputs from all previous steps. Data can be transformed using code steps (Node.js, Python, Go, Bash) or action components with built-in mapping. The platform supports nested data access (e.g., `steps.step1.data.user.email`) and provides utilities for common transformations (filtering, grouping, aggregation). Type information is inferred from component schemas, enabling IDE autocomplete.

Pipedream provides a library of 1000+ pre-built action components that encapsulate common operations on external services (e.g., 'send email via Gmail', 'create Jira issue', 'post Slack message'). Each action is a reusable component with built-in authentication, parameter validation, and error handling, located in components/{app_name}/actions/. Actions expose a schema-based property interface that generates UI forms automatically, allowing developers to configure them without writing API calls. The component system handles OAuth token refresh, rate limiting, and API versioning transparently.

Pipedream's component system supports dynamic properties that resolve at runtime based on user selections or previous step outputs, enabling cascading form fields (e.g., 'select account' → 'select project in that account'). Properties can define async functions that fetch options from APIs, implement conditional visibility based on other property values, and validate inputs against external schemas. The ComponentForm system in connect-react renders these properties as interactive forms, with real-time validation and dependency tracking.

Pipedream enables developers to define workflows as sequences of steps (event sources, actions, code) with conditional branching, error handlers, and step dependencies. Workflows are defined in YAML or through the web UI, with each step having access to outputs from previous steps via the `steps` object. The execution engine supports if/else branching based on step outputs, try/catch error handling, and step-level retry policies. Workflows run on Pipedream's serverless infrastructure with automatic scaling and no infrastructure management.

Pipedream provides a TypeScript-based SDK (@pipedream/sdk) for developers to build custom event sources and actions as reusable components. The SDK includes a type-safe component class system, property definition schema, authentication helpers, and a build system (tsup) that compiles TypeScript to CommonJS for execution. Components define their interface through property schemas that automatically generate UI forms, with built-in validation and type checking. The SDK supports composite projects and incremental builds for monorepo development.

Pipedream provides a Connect SDK (@pipedream/sdk) with BackendClient and BrowserClient implementations that enable developers to programmatically create, configure, and manage workflows outside the web UI. The BackendClient provides server-side APIs for workflow CRUD operations, component configuration, and execution management, while BrowserClient enables embedding Pipedream's component configuration UI (ComponentForm) in third-party applications. The SDK handles authentication token management, API communication, and state synchronization.

Pipedream supports scheduling workflows to run on a recurring basis using cron expressions or simple interval definitions (e.g., every 5 minutes, daily at 9 AM). Scheduled triggers are configured at the workflow level and execute on Pipedream's infrastructure without requiring external schedulers. The execution engine respects timezone settings and handles missed executions gracefully.

Pipedream automatically generates unique webhook URLs for each workflow, allowing external services to trigger workflows by sending HTTP POST requests. Webhooks support custom payload validation, header inspection, and request filtering. The webhook endpoint is publicly accessible and can receive JSON, form-encoded, or raw payloads. Pipedream handles webhook authentication through API keys or custom header validation.

Pipedream maintains a complete execution history for each workflow, including logs for every step, input/output data, error messages, and execution duration. Developers can inspect execution history through the web UI or API, with filtering and search capabilities. Logs include step-level details (e.g., API response codes, error stack traces) enabling debugging without access to external systems. Execution history is retained for a configurable period (typically 30 days on free tier).