apache / apache/airflow

Support disabling of dag_dir_list_interval

Open
#28,678 2 comments 0 reactions 1 assignee Claimed by @hussein-awala View on GitHub
area:core kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

2.5.0

### What happened

We have a unique situation with our Airflow deployment, in that we bake the DAGs into the image pre-deployment, meaning that a deployment is triggered every time there is a new DAG contribution.

During a recent performance troubleshooting, we have uncovered that the scheduler consumes much more memory than it should, and while looking for ways to optimize it, we came across the `dag_dir_list_interval` configuration. It's currently set to the default of 5 minutes. However, it seems like there is no possibility to disable it barring the possibility of setting it to a really high value like 86400.

### What you think should happen instead

It would be nice if the parameter supports a value, say `-1 `that disables the scan altogether.

### How to reproduce

Default configuration, the scan takes place every 5 minutes.

### Operating System

Debian GNU/Linux

### Versions of Apache Airflow Providers

_No response_

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

Airflow deployed with official helm chart on AKS.

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.