[Feature request] Add reverse run of DAGs
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
**Description of the problem**
A common workflow we have is running daily DAGs not dependent of past runs. When deployed, It need to catch up for many years before the runs are up to date and can be used by analysts.
The problem is that most of times, recent runs are more used/valuable than performing ETL of execution time 20 years ago. In practice we deploy the DAG two times, one for filling recent data and one for filling since inception.
**Feature requested**
A flag (`back_run=False`) on DAG class. If this flag is set to `True`, DAG runs are scheduled in reverse order.
E.g. consider we're on day `10` and in the middle of the process a new run for day `11` happens:
the DAG runs would be scheduled like `10 -> 9 -> 8 -> 11 -> 7 -> 6 -> 5` (assuming the run for `11` happened after running for day `8`). This flag can only be used if `depends_on_past=False`
Contributor guide
Assessment
This issue has not been assessed yet.