Add batching controls to `MessageQueueTrigger`
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Description
I would like to propose adding batching controls to `MessageQueueTrigger`.
Currently, a DAG triggered by a messaging-based asset may receive multiple events in `triggering_asset_events`, but this does not seem to be configurable in a general way at the `MessageQueueTrigger` level, even if some providers may support provider-specific behavior.
It would be useful to support, for example, a parameter to control the maximum batch size and another one to control the maximum wait time before flushing.
### Use case/motivation
This would make `MessageQueueTrigger` easier to use in cases where message batching needs to be bounded and more predictable.
### Related issues
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by locating the Python implementation of MessageQueueTrigger and tracing how messaging-based assets populate triggering_asset_events. Compare any provider-specific batching behavior, then define how configurable maximum batch size and maximum wait time should apply; done means both controls are supported consistently at the trigger level.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100