apache / apache/airflow

Increase of scheduler loop duration after version 2.10.5

Open
#51,212 1 comment 0 reactions 0 assignees View on GitHub
_eol_affected_version:2.11 area:core area:performance area:scheduler kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Apache Airflow version

2.11.0

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

We observed a significant increase in scheduler loop duration after attempting to upgrade from Airflow 2.10.4 to 2.10.5. Due to this issue, we reverted the upgrade. However, after attempting to migrate from 2.10.4 to 2.11.0, the problem persists.

- v2.10.4

![Image](https://github.com/user-attachments/assets/d3104834-6786-4ec7-a8c3-c1e6e2776841)

- v2.11.0
After upgrading to version 2.11.0 we noticed that the scheduler loop increased to ~18s (2x the old values) running the same dag which has >5,000 tasks.

![Image](https://github.com/user-attachments/assets/ae3d1fff-9d32-49f5-af0f-6b3276065bbc)

### What you think should happen instead?

I’ve reviewed the code changes between versions but haven’t been able to identify any specific modification that could explain this increase in scheduler loop duration. Nothing in the diffs stands out as an obvious cause for the performance degradation.

### How to reproduce

Using Airflow v2.10.4
• Create a DAG with a large number of tasks (>5,000).
• Trigger the DAG and monitor the scheduler_scheduler_loop_duration metric.

Using Airflow v2.11.0
• Repeat the same setup: using a DAG with >5,000 tasks.
• Trigger the DAG and monitor the same scheduler_scheduler_loop_duration metric.

Expected vs. Observed Behavior
• Expected: The scheduler loop duration should remain consistent or show minimal variation between versions.
• Observed: In v2.10.4, the loop duration typically remains around 9 seconds. In v2.11.0, we observed a significant increase in this duration (2x).

### Operating System

Debian GNU/Linux 12 (bookworm)

### Versions of Apache Airflow Providers

- 2.10.4

![Image](https://github.com/user-attachments/assets/bb653728-8fd9-4352-8f26-9eff74a85ce7)

- 2.11.0
![Image](https://github.com/user-attachments/assets/184e31ef-0399-471f-9270-00367e0216bc)

### Deployment

Other Docker-based deployment

### Deployment details

Docker (v28.0.4) deployment

### Anything else?

Observed after upgrade attempt.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Reproduce the comparison using Airflow 2.10.4 and 2.11.0 with a DAG containing more than 5,000 tasks, while monitoring the scheduler_scheduler_loop_duration metric. Compare scheduler behavior and relevant version changes; done means identifying the regression and restoring loop duration close to the 2.10.4 baseline.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.