apache / apache/airflow

Enable setting pool in the expand method of a dynamic task

Open
#33,657 4 comments 5 reactions 0 assignees View on GitHub
area:dynamic-task-mapping kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Description

Allow pools to be set dynamically for each dynamic mapped task index via the `expand` method as opposed to being limited to the current behaviour that only allows a static definition for all mapped tasks via the `partial` method.

### Use case/motivation

Pools are very useful for having control over resource usage on the backend systems that can oftentimes be overwhelmed by concurrent tasks triggered by Airflow. With dynamic mapped tasks, we can have a scenario where different mapped instances may trigger work in different backend servers and having the ability to set different pools comes in handy allowing control over the number of running tasks for each backend server separately. Dynamic tasks were a game changer allowing to dynamically define parts of the DAG, making it more like a first-class citizen that benefits from the other established Airflow features like pools is another step forward.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by tracing how dynamic task mapping handles expand and how the partial method currently applies a static pool. Define the behavior for selecting a pool per mapped task index, then verify that dynamic instances can use different pools while preserving existing static pool behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.