apache / apache/airflow

DeadlineAlert records not created for updated DAGs — SLA alerts silently stop firing after DAG re-serialization

Open
#72,471 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Apache Airflow version

3.2.2

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

When a DAG is updated (any code change that produces a new `serialized_dag` row), `DeadlineAlert` records are not created for the new `serialized_dag`. All subsequent `dag_run`s for that DAG silently produce no `Deadline` entries and never fire deadline callbacks.

**Root cause**

In `SerializedDagModel.write_dag()` (`airflow/models/serialized_dag.py`), when the deadline definition hasn't changed between versions, `_try_reuse_deadline_uuids()` returns the existing UUIDs and the code sets:

```python
# serialized_dag.py ~line 646-651
if deadline_uuid_mapping is not None:
# All deadlines matched — reuse the UUIDs to preserve hash.
# Clear the mapping since the alert rows already exist in the DB;
# no need to delete and recreate identical records.
dag.data["dag"]["deadline"] = existing_deadline_uuids
deadline_uuid_mapping = {}
```

Then in the new `serialized_dag` creation path (when `has_task_instances=True`):

```python
# line 734
cls._create_deadline_alert_records(new_serialized_dag, deadline_uuid_mapping)
```

Since `deadline_uuid_mapping = {}`, `_create_deadline_alert_records` returns immediately (`if not uuid_mapping: return`) — the new `serialized_dag` gets no `deadline_alert` rows.

When a `dag_run` is created, `definitions/dag.py` queries:

```sql
SELECT * FROM deadline_alert WHERE serialized_dag_id =
```

Finds nothing → no `Deadline` row inserted → triggerer never fires the callback.

The `{}` optimization is correct for the in-place UPDATE path (same `serialized_dag` row, existing `deadline_alert` records stay valid). It is wrong for the INSERT path (new row, no `deadline_alert` records exist for the new `serialized_dag_id`).

### What you think should happen instead?

`deadline_alert` records should be created for every new `serialized_dag` row, regardless of whether the deadline definition changed.

**Suggested fix**

In `_create_deadline_alert_records`, when the new `serialized_dag` path is taken (INSERT, not UPDATE), generate new UUIDs and create new `deadline_alert` records even when the deadline definition matches the previous version. The empty-mapping optimization should only apply to the in-place UPDATE path (lines 691–713).

### How to reproduce

1. Deploy a DAG with a `DeadlineAlert` — verify a `deadline_alert` record is created
2. Make any code change to the DAG file that produces a new `dag_version` (e.g. add a comment) without changing the deadline definition
3. Wait for git-sync to pick up the new version
4. Trigger a `dag_run` and wait past the deadline
5. No alert fires; the `deadline` table has no row for this run

### Operating System

N/A

### Versions of Apache Airflow Providers

N/A

### Deployment

Other 3rd-party Helm chart

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in airflow/models/serialized_dag.py around SerializedDagModel.write_dag(), _try_reuse_deadline_uuids(), and _create_deadline_alert_records(). Then trace the deadline_alert lookup in definitions/dag.py and reproduce the update-versus-new-serialized_dag paths described in the issue. Done means every new serialized_dag has matching deadline_alert rows while the in-place UPDATE optimization remains valid, and a subsequent dag_run creates a Deadline entry.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.