apache / apache/airflow

Deadline Alert triggered after dag failure

Open
#60,927 12 comments 0 reactions 0 assignees View on GitHub
affected_version:3.1 area:core area:deadline-alerts kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Apache Airflow version

Other Airflow 3 version (please specify below)

### If "Other Airflow 3 version" selected, which one?

3.1.5

### What happened?

I have a dag with deadline alert configured at 180 minutes, look below for code. The dag failed within 10 mins after a single retry but I still got the deadline alert 3 hrs later
`deadline=DeadlineAlert(
reference=DeadlineReference.DAGRUN_QUEUED_AT,
interval=timedelta(minutes=DEADLINE_ALERT_MINUTES),
callback=AsyncCallback(
send_rich_deadline_email,
kwargs={
"dag_id": f"{CLIENT_NAME}_{SERVER_NAME}_{JOB_NAME}",
"emails": EMAIL_ALERTS,
"deadline_min": DEADLINE_ALERT_MINUTES,
},`

### What you think should happen instead?

I should not have got a deadline alert. The issue is I have 4 dags that failed similarly but only one issued a deadline alert

### How to reproduce

Create a dag with the following, fail it before the deadline
`deadline=DeadlineAlert(
reference=DeadlineReference.DAGRUN_QUEUED_AT,
interval=timedelta(minutes=DEADLINE_ALERT_MINUTES),
callback=AsyncCallback(
send_rich_deadline_email,
kwargs={
"dag_id": f"{CLIENT_NAME}_{SERVER_NAME}_{JOB_NAME}",
"emails": EMAIL_ALERTS,
"deadline_min": DEADLINE_ALERT_MINUTES,
}`

### Operating System

RHEL 9.5

### Versions of Apache Airflow Providers

Package Name | Version | Description
apache-airflow-providers-common-compat | 1.11.0 | Common Compatibility Provider - providing compatibility code for previous Airflow versions
apache-airflow-providers-common-io | 1.7.0 | Common IO Provider
apache-airflow-providers-common-sql | 1.30.2 | Common SQL Provider https://en.wikipedia.org/wiki/SQL
apache-airflow-providers-fab | 3.1.1 | Flask App Builder https://flask-appbuilder.readthedocs.io/
apache-airflow-providers-postgres | 6.5.1 | PostgreSQL https://www.postgresql.org/
apache-airflow-providers-smtp | 2.4.1 | Simple Mail Transfer Protocol (SMTP) https://tools.ietf.org/html/rfc5321
apache-airflow-providers-standard | 1.10.2 | Airflow Standard Provider

### Deployment

Virtualenv installation

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the DAG's DeadlineAlert configuration using DeadlineReference.DAGRUN_QUEUED_AT and reproduce the failure-before-deadline case on Airflow 3.1.5. Trace how the deadline is handled after the DAG run fails, and consider the issue done when a failed run no longer produces an unexpected alert while the intended deadline alert behavior remains covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.