apache / apache/hudi

Rabbit MQ integration

Open
#14,021 2 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

**What the feature achieves:**
An integration with rabbit MQ, similar to the Kafka Connect one, that allows users to easily write the data that goes through the queue to the data lake.

**Why this feature is needed:**
I have many realtime systems that currently use rabbit MQ to pass data between different parts of the system. I want to store the data in my datalake as easily as I store the data that goes through by kafka message queues.

### User Experience

**How users will use this feature:**
- API changes
A new ingestion API, similar to the kafka-connect ingetion API (https://github.com/apache/hudi/tree/master/hudi-kafka-connect)

### Hudi RFC Requirements

**RFC PR link:** (if applicable)

**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? Yes
- Does this change storage format? No
I don't know if this feature need an RFC

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the hudi-kafka-connect integration referenced in the issue and compare its ingestion API with the requested RabbitMQ integration. Clarify the public API, RFC requirement, supported RabbitMQ behavior, and acceptance criteria before implementation; the feature is done when RabbitMQ queue data can be written to Hudi's data lake through a documented ingestion path.

Written by the indexing model from the issue text.

Assessment

Tech stack
rabbitmq
Domain
api, data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.