apache / apache/airflow

DAG-Level Automatic Retries Based on Terminal Task Status

Open
#60,866 7 comments 0 reactions 0 assignees View on GitHub
area:core kind:feature needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

Airflow currently supports retries at the task level, but there is no native way to automatically retry an entire DAG run from the beginning based on the outcome of a terminal (leaf) task or overall DAG run status.

Proposed behavior: a DAG-level retry mechanism that allows users to,
- Configure automatic retries of the entire DAG when one or more designated terminal tasks fail (or based on final DAG run state)
- Specify the number of DAG-level retries and retry delay
- Re-run all tasks from the start, rather than only retrying failed tasks (simulating a clearing of the run)
- Track DAG-level retry attempts separately from task-level retries

### Use case/motivation

In some workflows, a failure at the end of the DAG should indicate that upstream tasks must be re-executed from the very start of the run. Currently, achieving this requires manual clearing of task instances or custom logic, like calling the REST API to perform the clearing function for you within a callback of the leaf task.

For example, a DAG ends with a validation or publish task. If this task fails, earlier extraction or transformation steps must be re-run to ensure consistency. An automatic DAG-level retry would remove the need for manual intervention/custom logic.

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.