apache / apache/airflow

Airflow stops scheduling dagruns for other dags if stale dags are not deactivated

Open
#60,764 2 comments 0 reactions 0 assignees View on GitHub
area:core area:scheduler kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Apache Airflow version

Other Airflow 3 version (please specify below)

### If "Other Airflow 3 version" selected, which one?

3.0.6

### What happened?

When stale dags are not deactivated, then airflow scheduler gives errors like below. Eventually, the number of dagruns created by airflow is reduced and the overall throughput decreases.

### What you think should happen instead?

Airflow scheduler should ignore the dags that is not in serialized table and create dagruns for others.

### How to reproduce

Create a 2.x airflow environment with 10+ dags. Make sure the multiple dagruns can run concurrently for all dags. Upgrade airflow to 3.0.6 and at the same time remove atleast 5 dags during the upgrade.
Airflow will eventually stops scheduling dagruns for the other dags

### Operating System

Debian GNU/Linux 12 (bookworm)

### Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.12.2
apache-airflow-providers-cncf-kubernetes==10.8.2
apache-airflow-providers-common-compat==1.7.3
apache-airflow-providers-common-io==1.6.2
apache-airflow-providers-common-sql==1.27.5
apache-airflow-providers-fab==2.4.1
apache-airflow-providers-ftp==3.13.2
apache-airflow-providers-http==5.3.3
apache-airflow-providers-imap==3.9.2
apache-airflow-providers-postgres==6.2.3
apache-airflow-providers-smtp==2.2.0
apache-airflow-providers-sqlite==4.1.2
apache-airflow-providers-standard==1.6.0

### Deployment

Other

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.