Pipedream

Pipedream executes serverless workflows triggered by events (webhooks, schedules, manual triggers) with support for Node.js, Python, Go, and Bash code steps. Each workflow step runs in isolated containers with automatic dependency management and state passing between steps via JSON serialization. The platform handles infrastructure provisioning, scaling, and execution logging without requiring users to manage servers or containerization.

Pipedream maintains a monorepo of 1000+ open-source integration components (actions and event sources) for popular SaaS platforms, APIs, and services. Each component is a TypeScript/JavaScript module that encapsulates authentication, API interaction, and data transformation logic. Components are published to npm and can be discovered, configured, and used in workflows via the Pipedream UI or programmatically via the SDK, with automatic schema validation and property binding.

Pipedream supports scheduling workflows to run on a recurring basis using cron expressions (e.g., `0 0 * * *` for daily at midnight). Scheduled workflows are triggered by Pipedream's internal scheduler, which manages timing, retry logic, and execution history. The system supports multiple timezones and handles daylight saving time transitions automatically. Scheduled workflows can be paused, resumed, or modified without redeploying the workflow.

Pipedream allows developers to write custom code steps in Node.js, Python, Go, or Bash, with automatic dependency management via package managers (npm for Node.js, pip for Python, etc.). Each code step runs in an isolated container with access to the `steps` context object and environment variables. The platform handles runtime provisioning, dependency installation, and execution sandboxing, with support for importing npm packages, Python libraries, and system utilities.

Pipedream workflows can be version-controlled via Git, with workflow definitions stored as YAML or JSON files in a GitHub repository. The platform supports automatic deployment of workflows on Git push (via GitHub Actions or webhooks), enabling CI/CD workflows for workflow changes. Developers can review workflow changes via pull requests, with automated validation (linting, type checking) before merging. The system maintains a deployment history with rollback capability.

Pipedream provides built-in error handling and retry mechanisms for workflow steps. Failed steps can be configured to retry automatically with exponential backoff (e.g., 1s, 2s, 4s, 8s delays between retries). Developers can define custom error handlers that catch exceptions and decide whether to retry, skip, or fail the entire workflow. The system captures error stack traces and makes them available in logs for debugging.

Pipedream's component system supports dynamic property definitions where field visibility, validation, and available options depend on values of other fields. Properties are defined as TypeScript objects with async resolvers that fetch options from APIs (e.g., list of Slack channels, Google Sheet tabs) at configuration time. The system handles caching, error handling, and UI re-rendering when dependent properties change, enabling complex multi-step configuration flows without custom code.

Pipedream provides two mechanisms for ingesting events from external services: instant webhooks (services push events to Pipedream URLs) and polling sources (Pipedream periodically fetches data from APIs). Event sources are components that emit structured events into workflows, with built-in deduplication, error handling, and state management. The platform handles webhook URL generation, signature verification, and automatic retry logic, while polling sources manage cursor-based pagination and timestamp tracking to avoid duplicate processing.

The Pipedream Connect SDK provides TypeScript/JavaScript APIs for programmatically building, configuring, and embedding workflows outside the Pipedream UI. The SDK includes BackendClient (server-side API for workflow management) and BrowserClient (client-side UI components for embedding workflow builders in web apps). Developers can use the SDK to create custom workflow builders, embed component configuration forms in their own applications, and manage authentication tokens for multi-tenant scenarios.

Pipedream's ComponentForm system renders configuration UIs for workflow steps based on TypeScript property definitions. The system handles schema validation, error display, conditional field rendering, and custom input types (text, select, multiselect, code editor, etc.). Forms support theming via CSS variables and can be embedded in external applications via the Connect React package, with automatic handling of required fields, default values, and async property resolution.

Pipedream's Connect API provides server-side endpoints for managing user authentication, token generation, and workspace isolation in multi-tenant scenarios. The API handles OAuth token refresh, credential encryption, and user session management, allowing SaaS platforms to authenticate users and generate short-lived tokens for embedding Pipedream UIs. The system supports both user-level and workspace-level authentication with automatic token expiration and refresh logic.

Pipedream's component development system uses TypeScript with a custom build pipeline (tsup-based) that compiles components to CommonJS and ES modules with automatic dependency bundling and tree-shaking. The monorepo structure supports incremental builds via composite TypeScript projects, enabling fast development iteration. Components are published to npm as individual packages, with automatic version management and validation via CI/CD pipelines that check for breaking changes, missing documentation, and test coverage.

Pipedream workflows maintain state across steps by serializing step outputs to JSON and passing them to downstream steps as context. Each step receives a `steps` object containing outputs from all previous steps, enabling data chaining without explicit variable passing. The system handles JSON serialization/deserialization automatically, with support for complex objects, arrays, and nested structures. State is ephemeral (lost after workflow execution) unless explicitly persisted to external storage (databases, files, etc.).

Pipedream provides detailed execution logs for each workflow run, with step-level granularity showing inputs, outputs, duration, and errors for each step. Logs are stored in the Pipedream platform and accessible via the UI and API, with filtering by step, status, and timestamp. The system captures console output (console.log, console.error) from code steps and includes stack traces for errors, enabling developers to debug workflows without local debugging tools.