DeadlineAlert records not created for updated DAGs — SLA alerts silently stop firing after DAG re-serialization
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Apache Airflow version
3.2.2
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
When a DAG is updated (any code change that produces a new `serialized_dag` row), `DeadlineAlert` records are not created for the new `serialized_dag`. All subsequent `dag_run`s for that DAG silently produce no `Deadline` entries and never fire deadline callbacks.
**Root cause**
In `SerializedDagModel.write_dag()` (`airflow/models/serialized_dag.py`), when the deadline definition hasn't changed between versions, `_try_reuse_deadline_uuids()` returns the existing UUIDs and the code sets:
```python
# serialized_dag.py ~line 646-651
if deadline_uuid_mapping is not None:
# All deadlines matched — reuse the UUIDs to preserve hash.
# Clear the mapping since the alert rows already exist in the DB;
# no need to delete and recreate identical records.
dag.data["dag"]["deadline"] = existing_deadline_uuids
deadline_uuid_mapping = {}
```
Then in the new `serialized_dag` creation path (when `has_task_instances=True`):
```python
# line 734
cls._create_deadline_alert_records(new_serialized_dag, deadline_uuid_mapping)
```
Since `deadline_uuid_mapping = {}`, `_create_deadline_alert_records` returns immediately (`if not uuid_mapping: return`) — the new `serialized_dag` gets no `deadline_alert` rows.
When a `dag_run` is created, `definitions/dag.py` queries:
```sql
SELECT * FROM deadline_alert WHERE serialized_dag_id =
```
Finds nothing → no `Deadline` row inserted → triggerer never fires the callback.
The `{}` optimization is correct for the in-place UPDATE path (same `serialized_dag` row, existing `deadline_alert` records stay valid). It is wrong for the INSERT path (new row, no `deadline_alert` records exist for the new `serialized_dag_id`).
### What you think should happen instead?
`deadline_alert` records should be created for every new `serialized_dag` row, regardless of whether the deadline definition changed.
**Suggested fix**
In `_create_deadline_alert_records`, when the new `serialized_dag` path is taken (INSERT, not UPDATE), generate new UUIDs and create new `deadline_alert` records even when the deadline definition matches the previous version. The empty-mapping optimization should only apply to the in-place UPDATE path (lines 691–713).
### How to reproduce
1. Deploy a DAG with a `DeadlineAlert` — verify a `deadline_alert` record is created
2. Make any code change to the DAG file that produces a new `dag_version` (e.g. add a comment) without changing the deadline definition
3. Wait for git-sync to pick up the new version
4. Trigger a `dag_run` and wait past the deadline
5. No alert fires; the `deadline` table has no row for this run
### Operating System
N/A
### Versions of Apache Airflow Providers
N/A
### Deployment
Other 3rd-party Helm chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in airflow/models/serialized_dag.py around SerializedDagModel.write_dag(), _try_reuse_deadline_uuids(), and _create_deadline_alert_records(). Then trace the deadline_alert lookup in definitions/dag.py and reproduce the update-versus-new-serialized_dag paths described in the issue. Done means every new serialized_dag has matching deadline_alert rows while the in-place UPDATE optimization remains valid, and a subsequent dag_run creates a Deadline entry.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100