apache / apache/rocketmq-connect

message synchronization support up to the queue level

Open
#137 4 comments 0 reactions 0 assignees View on GitHub
discuss
Dominant language
Java
Stars
141
Forks
120
PR merge metrics
No merged PRs in 30d

Description

Now message synchronization is only supported to the topic level. The runtime does not specify the target queue when sending the message, so the queue after the message is sent to the target topic is random. This causes the messages of the source topic to be inconsistently queued in the target topic, which will cause confusion in message consumption after synchronizing the ConsumerOffset. Therefore, you can encapsulate SourceTask and SinkTask through WorkerDirectTask, specify the queue where the source message is located in SinkTask, and send the source message to the specified queue of the target topic through SinkTask. Want to add a SinkTask to RmqSourceReplicator to solve this problem, what do you think?

In this way, the synchronization of messages at the topic level can be solved by the current configuration, or the synchronization at the queue level can be solved by configuring WorkerDirectTask.

现在消息同步只支持到topic级别,发送消息时runtime并没有指定目标队列,所以消息发送到目标topic后的队列是随机的。这样就造成了源topic的消息在目标topic中队列不一致,这样在同步ConsumerOffset后会造成消息消费的错乱。因此可以通过WorkerDirectTask封装SourceTask和SinkTask,在SinkTask中指定源消息所在的队列,通过SinkTask将源消息发送到目标topic的指定队列。想为RmqSourceReplicator添加一个SinkTask去解决这个问题,大家怎么认为呢。

这样,既可以通过当前的配置解决消息在topic级别之间的同步,也可以通过配置WorkerDirectTask解决queue级别的同步。

Contributor guide

Open the contributing guide

Research direction

Start by reading RmqSourceReplicator and the existing SourceTask and SinkTask flow, then examine how WorkerDirectTask is intended to wrap them. Define how the target queue is selected from the source message while preserving current topic-level synchronization; done means queue-level synchronization can preserve queue alignment without breaking the existing configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.