apache / apache/rocketmq-flink

[Feature][RIP-74] Rocketmq-flink dynamic load balancing

Open
#125 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
174
Forks
104
PR merge metrics
No merged PRs in 30d

Description

In the original version, rocketmq-flink's queue allocation algorithm could not be averaged and could exacerbate unevenness when reduced. When the online environment experiences frequent scale-down, the situation becomes worse, and if you want to alleviate this situation, you can only restart the job, but cause large pauses. Therefore, we need to implement runtime dynamic allocation of queues.

RIP-docs: https://shimo.im/docs/RKAWMOEYgXfM88q8

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked RIP-docs design document, then trace the existing RocketMQ-Flink queue allocation logic in the source. Define how queue ownership changes during runtime scale-down and verify that rebalancing completes without requiring a job restart or causing large pauses.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.