apache / apache/airflow

AIP-82: Add common messaging system providers

Open
#52,712 27 comments 2 reactions 0 assignees View on GitHub
AIP-82 good first issue kind:feature kind:meta provider:amazon provider:google provider:microsoft-azure provider:redis
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Body

related: #49938

## Why

Several users have pointed out that the Event-Driven Scheduling feature introduced in Airflow 3 currently supports only Amazon SQS.

So I add Kafka common message queue provider in #49938, but there are still quite a few common providers that should be useful for airflow users to implement Event-Driven workflows.

## What

Here are some common messaging systems:

- [ ] Amazon Kinesis https://github.com/apache/airflow/pull/71135
- [ ] Amazon MSK https://github.com/apache/airflow/pull/69000
- [x] Amazon SQS
- [x] Apache Kafka
- [x] Azure Message Bus
- [x] Trigger: https://github.com/apache/airflow/pull/53356
- [x] CommonMessageQueue Interface
- [x] Google PubSub https://github.com/apache/airflow/pull/54494
- [x] Redis https://github.com/apache/airflow/pull/52917

reference for implementation: https://github.com/apache/airflow/pull/49938/files

### Committer

- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the implementation files in the referenced #49938/files comparison and the existing provider work linked in the checklist. Focus on the unchecked Amazon Kinesis and Amazon MSK entries, and use the completed providers to determine the expected trigger and CommonMessageQueue coverage. Done means the selected providers are implemented consistently and their checklist status is updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, azure, google-cloud, kafka, python, redis
Domain
backend, cloud, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.