apache / apache/airflow

Conditional Dataset(Asset) Updates Are Grouped Together Instead of Creating Distinct DAG Runs

Open
#53,896 5 comments 0 reactions 0 assignees View on GitHub
_eol_affected_version:2.10 area:core kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

Other Airflow 2 version (please specify below)

### If "Other Airflow 2 version" selected, which one?

2.10.5

### What happened?

I have a DAG with conditional dataset scheduling.
Image

Instead of creating a dag run per dataset update, sometimes airflow will group the events together (if they occur close enough)
Image

### What you think should happen instead?

I think there should be a distinct dag run per dataset update

### How to reproduce

Create a DAG with conditional scheduling

Create 2 dataset events at the same time that are consumed by the dag we created

### Operating System

linux

### Versions of Apache Airflow Providers

_No response_

### Deployment

Other Docker-based deployment

### Deployment details

_No response_

### Anything else?

Anytime there are dataset updates that happen at the same time or very close together

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the behavior in Apache Airflow 2.10.5 with conditional dataset scheduling and two dataset events occurring at the same time or close together. The issue does not name an implementation file or test; investigate the scheduler's dataset-event handling and verify that separate updates produce distinct DAG runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.