apache / apache/airflow

deferred / deferrable task are not take in account by max_active_tasks

Open
#40,528 10 comments 8 reactions 0 assignees View on GitHub
_eol_affected_version:2.9 area:async-operators area:Triggerer kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

2.9.2

### What happened?

If you set max_active_tasks, it do not limit the number of concurrent deferred tasks.

It's possible to restrict the number of concurrent deferred tasks with a pool but I want limit the number of tasks run by a dag and keep the possibility for every task to use a specific pool ( we can only use 1 pool by task in airflow )

### What you think should happen instead?

max_active_tasks should take in account the deferred task or we should create a new boolean setting

`max_active_tasks_include_deferred` like what was done for the pools

### How to reproduce

```python
from datetime import datetime, timezone, timedelta

from airflow.sensors.time_sensor import TimeSensorAsync
from pendulum import today
from airflow import DAG

with DAG(
dag_id='example',
schedule_interval='0 0 * * *',
max_active_tasks=10,
start_date=today("UTC").add(days=-1)):
for i in range(100):
TimeSensorAsync(task_id=f"a_{i}", target_time=(datetime.now(tz=timezone.utc) + timedelta(minutes=10)).time())

```

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by running the provided DAG with max_active_tasks=10 and TimeSensorAsync tasks, then inspect how the scheduler counts deferred tasks and how pool limits handle them. Done means deferred tasks are included in max_active_tasks, or an agreed max_active_tasks_include_deferred setting provides that behavior, with the concurrency behavior covered by regression tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.