apache / apache/airflow

[Feature request] Add reverse run of DAGs

Open
#9,176 4 comments 6 reactions 0 assignees View on GitHub
area:core kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

**Description of the problem**

A common workflow we have is running daily DAGs not dependent of past runs. When deployed, It need to catch up for many years before the runs are up to date and can be used by analysts.

The problem is that most of times, recent runs are more used/valuable than performing ETL of execution time 20 years ago. In practice we deploy the DAG two times, one for filling recent data and one for filling since inception.

**Feature requested**

A flag (`back_run=False`) on DAG class. If this flag is set to `True`, DAG runs are scheduled in reverse order.

E.g. consider we're on day `10` and in the middle of the process a new run for day `11` happens:
the DAG runs would be scheduled like `10 -> 9 -> 8 -> 11 -> 7 -> 6 -> 5` (assuming the run for `11` happened after running for day `8`). This flag can only be used if `depends_on_past=False`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.