AIP-82: Add common messaging system providers
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Body
related: #49938
## Why
Several users have pointed out that the Event-Driven Scheduling feature introduced in Airflow 3 currently supports only Amazon SQS.
So I add Kafka common message queue provider in #49938, but there are still quite a few common providers that should be useful for airflow users to implement Event-Driven workflows.
## What
Here are some common messaging systems:
- [ ] Amazon Kinesis https://github.com/apache/airflow/pull/71135
- [ ] Amazon MSK https://github.com/apache/airflow/pull/69000
- [x] Amazon SQS
- [x] Apache Kafka
- [x] Azure Message Bus
- [x] Trigger: https://github.com/apache/airflow/pull/53356
- [x] CommonMessageQueue Interface
- [x] Google PubSub https://github.com/apache/airflow/pull/54494
- [x] Redis https://github.com/apache/airflow/pull/52917
reference for implementation: https://github.com/apache/airflow/pull/49938/files
### Committer
- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Contributor guide
Research direction
Start by reviewing the implementation files in the referenced #49938/files comparison and the existing provider work linked in the checklist. Focus on the unchecked Amazon Kinesis and Amazon MSK entries, and use the completed providers to determine the expected trigger and CommonMessageQueue coverage. Done means the selected providers are implemented consistently and their checklist status is updated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, azure, google-cloud, kafka, python, redis
- Domain
- backend, cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100