Asset event scheduling - introduce max_asset_events parameter
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
Currently, when a DAG is triggered by asset events, Airflow's scheduler consumes all available asset events from the queue and creates a single DAG run that processes all these events together ([code](https://github.com/apache/airflow/blob/3.0.4/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L1613-L1641)).
This feature request proposes adding a configurable parameter (eg. max_asset_events) that allows DAG developers to control how many asset events are consumed per DAG run.
### Use case/motivation
This feature request is primarily motivated by the need to simplify migration from Airflow 2.x DAGs that were previously triggered by SQS queue events and designed to process one message at a time (as `SqsSensor` allowed to configure `max_messages=1` parameter).
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with the asset-event scheduling logic in airflow-core/src/airflow/jobs/scheduler_job_runner.py at the linked lines, then compare the requested behavior with SqsSensor's max_messages=1 configuration. Define how max_asset_events would be exposed to DAG developers and confirm that each DAG run consumes no more than the configured number of events.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100