apache / apache/airflow

DAG Runs from a single DAG can prevent scheduler from seeing other DAG's runs

Open
#49,508 33 comments 1 reaction 1 assignee Claimed by @uranusjr View on GitHub
_eol_affected_version:2.10 affected_version:main_branch area:core area:scheduler kind:bug priority:medium
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

2.10.5

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

Two DAGs each receive a large batch of DAG Runs. The number of runs for each DAG exceeds `max_dagruns_per_loop_to_schedule`. Each DAG run is very short, shorter than the heartrate of this Airflow deployment. Both DAGs have a max_active_runs that is far less than dagruns_per_loop.

So: max_active_runs < max_dagruns_per_loop_to_schedule < number of queued DAG runs.

Each scheduler loop, there are a very small number of DAG Run "slots" for the first DAG, so the check `coalesce(running_drs.c.num_running, text("0")) < coalesce(Backfill.max_active_runs, DagModel.max_active_runs),` does not apply. But then all the DAG runs that are considered are from the first DAG. So Second DAG effectively has to wait for nearly all of First DAG's runs to complete before any of its runs are moved from queued to running.

### What you think should happen instead?

I think the "most correct" thing to do is to change the global yes/no for a DAG being included in the check on the basis of max_active_runs to some kind of limit on the number for that DAG that can be included. I can't see a good way to do this in SQL but others may have insight.

Alternatively, because this is predominantly a problem when a single DAG dominates the scheduler's attention, we could add an explicit check to see if the result of the DAG run query contains only the a single DAG, and if so re-run the query with that DAG excluded.

### How to reproduce

1. Create two DAGs with a single, simple task.
2. Set max_active_runs=100
3. Set max_dagruns_per_loop_to_schedule=2000
4. Start 5000 Runs of the first DAG
5. Start 5000 Runs of the second DAG
6. Hard to reproduce: keep the heartrate of the scheduler low enough that Runs complete within one scheduler loop.

### Operating System

Debian GNU/Linux 12 (bookworm)

### Versions of Apache Airflow Providers

_No response_

### Deployment

Astronomer

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.