apache / apache/airflow

Airflow DAG version keeps increasing

Open
#58,310 10 comments 0 reactions 0 assignees View on GitHub
area:core area:serialization kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

Other Airflow 2/3 version (please specify below)

### If "Other Airflow 2/3 version" selected, which one?

3.1.0

### What happened?

We have a DAG which each time it is triggered would create 3 DAG versions. Question is why is this happening , as the impact is that the serialized_dag table is growing and would require clean up

Image

### What you think should happen instead?

would like to understand how does DAG version being triggered , we would not like to have multiple versions

### How to reproduce

in this example i am not sure what triggered the DAG versions being increased , but basically we are using some custom operators and looping through a list to generate python operators

### Operating System

RHEL

### Versions of Apache Airflow Providers

apache-airflow==3.1.0rc2
apache-airflow-core==3.1.0rc2
apache-airflow-providers-amazon==9.14.0
apache-airflow-providers-celery==3.12.3
apache-airflow-providers-cncf-kubernetes==10.5.0
apache-airflow-providers-common-compat==1.7.4
apache-airflow-providers-common-io==1.6.3
apache-airflow-providers-common-messaging==2.0.0
apache-airflow-providers-common-sql==1.28.1
apache-airflow-providers-docker==4.4.3
apache-airflow-providers-elasticsearch==6.3.3
apache-airflow-providers-fab==2.4.3
apache-airflow-providers-ftp==3.13.2
apache-airflow-providers-git==0.0.8
apache-airflow-providers-google==18.0.0
apache-airflow-providers-grpc==3.8.2
apache-airflow-providers-hashicorp==4.3.2
apache-airflow-providers-http==5.3.4
apache-airflow-providers-microsoft-azure==12.7.1
apache-airflow-providers-microsoft-mssql==4.3.2
apache-airflow-providers-mysql==6.3.4
apache-airflow-providers-odbc==4.10.2
apache-airflow-providers-openlineage==2.7.1
apache-airflow-providers-postgres==6.3.0
apache-airflow-providers-redis==4.3.1
apache-airflow-providers-sendgrid==4.1.3
apache-airflow-providers-sftp==5.4.0
apache-airflow-providers-slack==9.3.0
apache-airflow-providers-smtp==2.2.1
apache-airflow-providers-snowflake==6.5.4
apache-airflow-providers-ssh==4.1.4
apache-airflow-providers-standard==1.8.0
apache-airflow-task-sdk==1.1.0rc2

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

_No response_

### Anything else?

this issue occurs everytime DAG is triggered ( scheduled daily at midnight ) and will generate 3 versions

Image

Image

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the DAG definition, its custom operators, and the loop that generates Python operators; reproduce the daily trigger while observing changes in the serialized_dag table. Done means identifying why one trigger creates three versions and documenting or correcting that behavior so unexpected version growth no longer occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.