apache / apache/airflow

Allow multiple pools for one task

Open
#13,975 9 comments 38 reactions 0 assignees View on GitHub
airflow3.0:candidate area:scheduler kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

Hello!

Description of feature:

I think it would be helpful to allow for multiple pools in one task. Currently, the `pool` argument for any class inheriting from `BaseOperator` is of type `string`, and thus only allows one pool to be entered. I believe it would be useful to allow to set multiple pools for one task, meaning allowing the argument `pool` to be a `list` of `string` instead of just one `string`. This would mean that a task would have to wait on a spot to be available in every one of the pools it declares, instead of in the only one pool it declares, and this would mean that a task would take up spots in every one of the pools it declares, instead of in only the one pool it declares.

Use case:

I have some tasks that require multiple resources. I cannot split the tasks into separate tasks each requiring one resource, since the tasks need the two (or more) resources at once to complete their assignment. I also have some tasks only requiring one of the resources, so I can't create a pool for both resources.
Example:
Task 1 requires resource A and B
Task 2 requires resource A
Task 3 requires resource B
Resource A can only have 4 connections.
Resource B can only have 16 connections.
I would need to have task 1 be in pool A and pool B, and this is not possible today since I can only specify one pool.

What would I want to happen?

Allow multiple pools in task creation. I looked into airflow source code, and it looks like the assumption that we only have one pool is deep into SQL, so I cannot just easily fork airflow and add this feature, so the change is not small and I do not have enough airflow understanding to make this change.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how BaseOperator's pool argument is represented and enforced, then follow the SQL assumptions the issue identifies. Review the scheduler's pool-waiting and reservation flow. Done means a task can declare multiple pools and waits for and occupies capacity in every declared pool.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.