Increase of scheduler loop duration after version 2.10.5
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Apache Airflow version
2.11.0
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
We observed a significant increase in scheduler loop duration after attempting to upgrade from Airflow 2.10.4 to 2.10.5. Due to this issue, we reverted the upgrade. However, after attempting to migrate from 2.10.4 to 2.11.0, the problem persists.
- v2.10.4

- v2.11.0
After upgrading to version 2.11.0 we noticed that the scheduler loop increased to ~18s (2x the old values) running the same dag which has >5,000 tasks.

### What you think should happen instead?
I’ve reviewed the code changes between versions but haven’t been able to identify any specific modification that could explain this increase in scheduler loop duration. Nothing in the diffs stands out as an obvious cause for the performance degradation.
### How to reproduce
Using Airflow v2.10.4
• Create a DAG with a large number of tasks (>5,000).
• Trigger the DAG and monitor the scheduler_scheduler_loop_duration metric.
Using Airflow v2.11.0
• Repeat the same setup: using a DAG with >5,000 tasks.
• Trigger the DAG and monitor the same scheduler_scheduler_loop_duration metric.
Expected vs. Observed Behavior
• Expected: The scheduler loop duration should remain consistent or show minimal variation between versions.
• Observed: In v2.10.4, the loop duration typically remains around 9 seconds. In v2.11.0, we observed a significant increase in this duration (2x).
### Operating System
Debian GNU/Linux 12 (bookworm)
### Versions of Apache Airflow Providers
- 2.10.4

- 2.11.0

### Deployment
Other Docker-based deployment
### Deployment details
Docker (v28.0.4) deployment
### Anything else?
Observed after upgrade attempt.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Reproduce the comparison using Airflow 2.10.4 and 2.11.0 with a DAG containing more than 5,000 tasks, while monitoring the scheduler_scheduler_loop_duration metric. Compare scheduler behavior and relevant version changes; done means identifying the regression and restoring loop duration close to the 2.10.4 baseline.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100