deferred / deferrable task are not take in account by max_active_tasks
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
2.9.2
### What happened?
If you set max_active_tasks, it do not limit the number of concurrent deferred tasks.
It's possible to restrict the number of concurrent deferred tasks with a pool but I want limit the number of tasks run by a dag and keep the possibility for every task to use a specific pool ( we can only use 1 pool by task in airflow )
### What you think should happen instead?
max_active_tasks should take in account the deferred task or we should create a new boolean setting
`max_active_tasks_include_deferred` like what was done for the pools
### How to reproduce
```python
from datetime import datetime, timezone, timedelta
from airflow.sensors.time_sensor import TimeSensorAsync
from pendulum import today
from airflow import DAG
with DAG(
dag_id='example',
schedule_interval='0 0 * * *',
max_active_tasks=10,
start_date=today("UTC").add(days=-1)):
for i in range(100):
TimeSensorAsync(task_id=f"a_{i}", target_time=(datetime.now(tz=timezone.utc) + timedelta(minutes=10)).time())
```
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by running the provided DAG with max_active_tasks=10 and TimeSensorAsync tasks, then inspect how the scheduler counts deferred tasks and how pool limits handle them. Done means deferred tasks are included in max_active_tasks, or an agreed max_active_tasks_include_deferred setting provides that behavior, with the concurrency behavior covered by regression tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100