apache / apache/airflow

Add an option to stop backfill automatically if one dag has failed

Open
#60,767 3 comments 0 reactions 0 assignees View on GitHub
area:backfill kind:feature needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

As a maintainer of a dag
I would like to stop the backfill automatically if one dag execution has failed
In order to avoid useless computation and logs

As a maintainer of a dag
I would like to add an option `Stop backfill if on dag has failed` in the `Trigger > Backfill` form, just below `Run backwards` option
In order to change backfill behaviour

Image

As a maintainer of a dag
I would like to resume backfill execution after failure
In order to continue backfill once the issue is resolved

### Use case/motivation

In our case, each dag execution depends on the previous one. Thus, during a backfill if one dag execution has failed all others will also fail.

Image

So, all dag will fail in cascade, causing useless computation and logs.

This new option `Stop backfill if on dag has failed` should stop backfill after the first dag execution failure.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.