apache / apache/airflow

Add batching controls to `MessageQueueTrigger`

Open
#63,924 2 comments 0 reactions 0 assignees View on GitHub
AIP-82 area:core kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

I would like to propose adding batching controls to `MessageQueueTrigger`.

Currently, a DAG triggered by a messaging-based asset may receive multiple events in `triggering_asset_events`, but this does not seem to be configurable in a general way at the `MessageQueueTrigger` level, even if some providers may support provider-specific behavior.

It would be useful to support, for example, a parameter to control the maximum batch size and another one to control the maximum wait time before flushing.

### Use case/motivation

This would make `MessageQueueTrigger` easier to use in cases where message batching needs to be bounded and more predictable.

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by locating the Python implementation of MessageQueueTrigger and tracing how messaging-based assets populate triggering_asset_events. Compare any provider-specific batching behavior, then define how configurable maximum batch size and maximum wait time should apply; done means both controls are supported consistently at the trigger level.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.