Data Pipelines

Python workflow orchestration — decorators for tasks/flows, retries, caching, scheduling.

Common Crawl

60/100Free

Largest open web crawl archive, foundation of all LLM training data.

Dataset

BioGPT Agent

Microsoft's AI agent for biomedical research.

Agent

Tecton

58/100Free

Enterprise real-time feature platform for production ML.

Platform

Stagehand

AI browser automation — natural language commands for web actions, built on Playwright.

Spring AI

63/100Free

AI framework for Spring/Java — portable LLM API, RAG pipeline, vector stores, function calling.

Kestra

Unified orchestration with declarative YAML.

CVAT

Open-source computer vision annotation tool.

Apify

Web scraping platform with 2,000+ ready-made scrapers.

Platform

Airbyte

Open-source ELT platform with 300+ connectors.

Soda

Data quality checks with human-readable SodaCL language.

Great Expectations

Data quality validation framework with declarative expectations.

Elementary

Open-source dbt-native data observability and anomaly detection.

DVC

Git for data and ML — version large files, experiment tracking, pipeline DAGs, remote storage.

dlt

Python data load tool with automatic schema inference.

Dagster

60/100Free

Data orchestration for ML — software-defined assets, type-checked IO, observability, modern Airflow alternative.

Crawl4AI

AI-optimized web crawler — clean markdown extraction, JS rendering, structured output for RAG.