Deadline Alert triggered after dag failure
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Apache Airflow version
Other Airflow 3 version (please specify below)
### If "Other Airflow 3 version" selected, which one?
3.1.5
### What happened?
I have a dag with deadline alert configured at 180 minutes, look below for code. The dag failed within 10 mins after a single retry but I still got the deadline alert 3 hrs later
`deadline=DeadlineAlert(
reference=DeadlineReference.DAGRUN_QUEUED_AT,
interval=timedelta(minutes=DEADLINE_ALERT_MINUTES),
callback=AsyncCallback(
send_rich_deadline_email,
kwargs={
"dag_id": f"{CLIENT_NAME}_{SERVER_NAME}_{JOB_NAME}",
"emails": EMAIL_ALERTS,
"deadline_min": DEADLINE_ALERT_MINUTES,
},`
### What you think should happen instead?
I should not have got a deadline alert. The issue is I have 4 dags that failed similarly but only one issued a deadline alert
### How to reproduce
Create a dag with the following, fail it before the deadline
`deadline=DeadlineAlert(
reference=DeadlineReference.DAGRUN_QUEUED_AT,
interval=timedelta(minutes=DEADLINE_ALERT_MINUTES),
callback=AsyncCallback(
send_rich_deadline_email,
kwargs={
"dag_id": f"{CLIENT_NAME}_{SERVER_NAME}_{JOB_NAME}",
"emails": EMAIL_ALERTS,
"deadline_min": DEADLINE_ALERT_MINUTES,
}`
### Operating System
RHEL 9.5
### Versions of Apache Airflow Providers
Package Name | Version | Description
apache-airflow-providers-common-compat | 1.11.0 | Common Compatibility Provider - providing compatibility code for previous Airflow versions
apache-airflow-providers-common-io | 1.7.0 | Common IO Provider
apache-airflow-providers-common-sql | 1.30.2 | Common SQL Provider https://en.wikipedia.org/wiki/SQL
apache-airflow-providers-fab | 3.1.1 | Flask App Builder https://flask-appbuilder.readthedocs.io/
apache-airflow-providers-postgres | 6.5.1 | PostgreSQL https://www.postgresql.org/
apache-airflow-providers-smtp | 2.4.1 | Simple Mail Transfer Protocol (SMTP) https://tools.ietf.org/html/rfc5321
apache-airflow-providers-standard | 1.10.2 | Airflow Standard Provider
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with the DAG's DeadlineAlert configuration using DeadlineReference.DAGRUN_QUEUED_AT and reproduce the failure-before-deadline case on Airflow 3.1.5. Trace how the deadline is handled after the DAG run fails, and consider the issue done when a failed run no longer produces an unexpected alert while the intended deadline alert behavior remains covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100