Conditional Dataset(Asset) Updates Are Grouped Together Instead of Creating Distinct DAG Runs
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.10.5
### What happened?
I have a DAG with conditional dataset scheduling.
Instead of creating a dag run per dataset update, sometimes airflow will group the events together (if they occur close enough)
### What you think should happen instead?
I think there should be a distinct dag run per dataset update
### How to reproduce
Create a DAG with conditional scheduling
Create 2 dataset events at the same time that are consumed by the dag we created
### Operating System
linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### Anything else?
Anytime there are dataset updates that happen at the same time or very close together
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by reproducing the behavior in Apache Airflow 2.10.5 with conditional dataset scheduling and two dataset events occurring at the same time or close together. The issue does not name an implementation file or test; investigate the scheduler's dataset-event handling and verify that separate updates produce distinct DAG runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100