apache / apache/airflow

Datasets: Allow time based force dependency when dataset is not ready.

Open
#30,974 15 comments 9 reactions 0 assignees View on GitHub
area:data-aware-scheduling kind:feature priority:low
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Body

**The current state:** If DAG is defined with 4 datasets. Airflow will wait for all of them to be ready before scheduling the DAG. This works well and serve use cases where all 4 datasets are curial and must be ready.

**The use case we don't currently handle:** It is common for datasets not to be equally important. Sometimes the core datasets are ready yet some minor ones are not (for example if one of the datasets is used as enrichment) in that case DAG author may want to define "grace period" which means how much time he is willing to continue to wait before DAG should be scheduled regardless if dataset is ready or not. With pipeline sometimes "good enough" is OK. The worst that can happen is that one major pipeline (which has wide downstream depended DAGs) is stuck on some minor dataset.

**Suggested ideas:**
1. Introduce the ability to skip dataset dependency check after grace period has passed.
2. Add `DatasetSensor`? That can be used as a workaround (minor dependencies can be set within the DAG and not be used with the dataset feature) I don't like this one so much but this is an option.

(Inspired by https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1682791720916639 )

### Committer

- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Research direction

Start by reading Airflow's dataset dependency and scheduling behavior, with the proposed DatasetSensor workaround as a comparison. Clarify how a grace period should interact with multiple datasets and what happens when it expires; done means the scheduler can proceed after the configured wait while preserving the existing all-datasets-ready behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.