apache / apache/airflow

Asset event scheduling - introduce max_asset_events parameter

Open
#55,956 7 comments 6 reactions 0 assignees View on GitHub
area:data-aware-scheduling kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Currently, when a DAG is triggered by asset events, Airflow's scheduler consumes all available asset events from the queue and creates a single DAG run that processes all these events together ([code](https://github.com/apache/airflow/blob/3.0.4/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L1613-L1641)).

This feature request proposes adding a configurable parameter (eg. max_asset_events) that allows DAG developers to control how many asset events are consumed per DAG run.

### Use case/motivation

This feature request is primarily motivated by the need to simplify migration from Airflow 2.x DAGs that were previously triggered by SQS queue events and designed to process one message at a time (as `SqsSensor` allowed to configure `max_messages=1` parameter).

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the asset-event scheduling logic in airflow-core/src/airflow/jobs/scheduler_job_runner.py at the linked lines, then compare the requested behavior with SqsSensor's max_messages=1 configuration. Define how max_asset_events would be exposed to DAG developers and confirm that each DAG run consumes no more than the configured number of events.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.