apache / apache/airflow

Allow dynamically set `pool_slots` for potentially heavy tasks.

Open
#35,803 6 comments 5 reactions 0 assignees View on GitHub
area:scheduler kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Description

Allow to set pool_slots for a task based on the result of another task.

One task may require different amount of resources for runtime depending on its inputs. However we can assume approximate resource requirements for a task based on its initial parameters. It would be nice to be able to dynamically set `pool_slots` based on the result of another task.

```python

@dag
def example_dag():

@task
def get_slots_required():
from lib import estimate_cost
data = get_data()
return 3 if estimate_cost(data) > 15 else 1

@task
def heavy_or_not_task():
...

slots = get_slots_required()
heavy_or_not_task.override(pool_slots=slots)()

dag_instance = example_dag()
```

### Use case/motivation

This feature expands capabilities of pool system in Airflow allowing more fine grained control over resources making system more stable and efficient.

### Related issues

https://github.com/apache/airflow/issues/33657

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the task decorator and the shown override(pool_slots=...) usage, then trace how pool_slots is consumed by Airflow's pool and scheduling system. Done means a task can receive pool_slots from another task's result while preserving correct resource allocation; the related issue may provide additional context.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.