Send task to queue in bulk - Celery Executor
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
**Description**
Hello,
I recently took care of CeleryExecutor. I managed to optimize the status retrieval by using bulk operations. Instead of fetching the status for each task using a separate query, one is sent for all tasks. This has accelerated this process more than 100 times in many cases.
https://github.com/apache/airflow/pull/7542
However, we still use single requests in many processes to send tasks to the queue. This is very effective because of network latency.
https://github.com/apache/airflow/blob/f1dc2e0b0e358582c1df0cc07a5cc95fa721dc44/airflow/executors/celery_executor.py#L196-L206
It would be nice if it could be done as a bulk request in a single request. For Redis, this means using Pipeline.
https://github.com/andymccurdy/redis-py#pipelines
Can it be done easily in Celery?
Best regards,
Kamil
Contributor guide
Assessment
This issue has not been assessed yet.