For batch-processing jobs, the strongest candidates are Apache Airflow for scheduled, multi-step batch workflows 10, Celery for distributed task queues processing large volumes of messages 7, and Dask for parallel batch computation in Python 9. BullMQ 2 and Python-rq 6 suit lighter background job queues on Redis, and Google's GKE batch/HPC tooling 1 covers running batch workloads on Kubernetes.

1. 1

   Apache Airflowapache.orgverified and signed

   Programmatically authors, schedules and monitors batch workflows in Python, with a web UI and an API for batch task and DAG run listings 10.

   WebsiteDocs

   official:API · Airflow API (Stable)API · Airflow API (Stable)

   for agents: get_task_instances_batch · get_task_instances_batch · get_dag_runs_batch

2. 2

   Celeryceleryq.devverified and signed

   Distributed task queue that processes large amounts of messages and background jobs across workers, with task scheduling support 7.

   WebsiteDocs

   for agents: queue · job · task

3. 3

   Daskdask.orgverified and signed

   Parallel computing library for Python that scales batch data processing across machines, with DataFrames built on pandas 9.

   WebsiteDocsGet started

   +2 community tools built for it

   for agents: task-scheduling · parallel · pandas

4. 4

   BullMQbullmq.ioverified and signed

   Open source message queue backed by Redis or PostgreSQL for running background jobs, available for Node.js, Python and other languages 2.

   WebsiteDocs

   +2 community tools built for it

   for agents: queues · python · bullmq

5. 5

   Google Cloud (GKE batch/HPC)google.comverified and signed

   Runs batch and HPC workloads on GKE using job queues and parallel processing 1.

   Website

   official:MCP server · mcp-securityAPI · Travel Partner APISDK · brotlilibrary · zerocopyCLI · googletest+1,236 community tools built for it

   for agents: gke-batch-hpc · dataproc · cloud-run-basics