apache / apache/airflow

Send task to queue in bulk - Celery Executor

Open
#8,854 6 comments 2 reactions 1 assignee Claimed by @hussein-awala View on GitHub
area:providers kind:feature provider:celery
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

**Description**

Hello,
I recently took care of CeleryExecutor. I managed to optimize the status retrieval by using bulk operations. Instead of fetching the status for each task using a separate query, one is sent for all tasks. This has accelerated this process more than 100 times in many cases.
https://github.com/apache/airflow/pull/7542
However, we still use single requests in many processes to send tasks to the queue. This is very effective because of network latency.
https://github.com/apache/airflow/blob/f1dc2e0b0e358582c1df0cc07a5cc95fa721dc44/airflow/executors/celery_executor.py#L196-L206
It would be nice if it could be done as a bulk request in a single request. For Redis, this means using Pipeline.
https://github.com/andymccurdy/redis-py#pipelines

Can it be done easily in Celery?

Best regards,
Kamil

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.