Allow dynamically set `pool_slots` for potentially heavy tasks.
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Description
Allow to set pool_slots for a task based on the result of another task.
One task may require different amount of resources for runtime depending on its inputs. However we can assume approximate resource requirements for a task based on its initial parameters. It would be nice to be able to dynamically set `pool_slots` based on the result of another task.
```python
@dag
def example_dag():
@task
def get_slots_required():
from lib import estimate_cost
data = get_data()
return 3 if estimate_cost(data) > 15 else 1
@task
def heavy_or_not_task():
...
slots = get_slots_required()
heavy_or_not_task.override(pool_slots=slots)()
dag_instance = example_dag()
```
### Use case/motivation
This feature expands capabilities of pool system in Airflow allowing more fine grained control over resources making system more stable and efficient.
### Related issues
https://github.com/apache/airflow/issues/33657
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by reviewing the task decorator and the shown override(pool_slots=...) usage, then trace how pool_slots is consumed by Airflow's pool and scheduling system. Done means a task can receive pool_slots from another task's result while preserving correct resource allocation; the related issue may provide additional context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100