apache / apache/airflow

add option to set DAG level Dataset outlets

Open
#39,105 10 comments 2 reactions 0 assignees View on GitHub
area:data-aware-scheduling kind:feature priority:low
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Description

add a new argument `outlets` to the DAG constructor with the same effect as the BaseOperator's outlets argument when the full dag run is successfull

### Use case/motivation

use case:
I'm currently maintaining several DAGs with "Dataset Scheduling", where each DAG handles a specific dataset and declares that it is ready for downstream DAGs by an EmptyOperator with an `outlets` value
this means I need to fan-in all the leafs of each DAG:
![image](https://github.com/apache/airflow/assets/48283294/311fb3e1-2116-4433-879b-eac55c9c8f17)

suggestion:
add an `outlets` argument to the `DAG` constructor to mark the Dataset(s) as ready when the dag run is successful

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start at the DAG constructor and compare the requested behavior with the existing BaseOperator outlets argument and Dataset Scheduling flow. Trace how a successful full DAG run marks outlets ready, then verify the behavior with the relevant existing scheduling tests; done means DAG-level outlets are emitted only after successful runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.