apache / apache/airflow

DAGs that have entires in `dag_version` but not in `serialized_dag` are never serialized

Open
#68,944 1 comment 0 reactions 0 assignees View on GitHub
area:core area:serialization kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Airflow Core

### Apache Airflow version

3.1.6 (the source code seems the same at 3.2.2)

### What happened and how to reproduce it?

in https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/models/serialized_dag.py in the method `write_dag`, you have the following block:
```
if dag_version and not has_task_instances:
# This is for dynamic DAGs that the hashes changes often. We should update
# the serialized dag, the dag_version and the dag_code instead of a new version
# if the dag_version is not associated with any task instances

# Use direct UPDATE to avoid loading the full serialized DAG
result = session.execute(
update(cls)
.where(cls.dag_version_id == dag_version.id)
.values(
{
cls._data: new_serialized_dag._data,
cls._data_compressed: new_serialized_dag._data_compressed,
cls.dag_hash: new_serialized_dag.dag_hash,
}
)
)

if getattr(result, "rowcount", 0) == 0:
# No rows updated - serialized DAG doesn't exist
return False
```
This allows the following scenario to happen - you have the dag in `dag_version` without task instances, but not in `serialized_dag` - in which case the dag is never serialized.

### What you think should happen instead?

the dag should be serialized if it is missing

### Operating System

_No response_

### Deployment

Other

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

Context:
I ran into this state while running a custom application on top of airflow. It is highly likely that it was caused by custom code and not airflow, but airflow doesn't seem to have any safeguards against entering the state, and it affected all instances of my application. I can't easily tell exactly how it happened exactly (we are in the middle of updating from airflow 2 to 3 and there are lots of changes)
I believe this fix is appropriate given that there should be multiple ways to reach this state e.g. by editing the db manually.
An alternative would be to try and add safeguards against getting into this state in the first place, but that would be way more complex than handling in this way + currently this is done at no additional cost as we already know whether the serialized dag exists or not by the old hash.
There should be no implications on backwards compatibility.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in airflow-core/src/airflow/models/serialized_dag.py and read the write_dag method, especially the direct UPDATE path for a dag_version without task instances. Reproduce or inspect the state where dag_version exists but serialized_dag does not, then verify that the missing DAG is serialized and that the existing update behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.