Retain instance sequence for table rebalance
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Currently when the table rebalance request is made, the list of instances will be sorted by the instance ids before the instance partition is formed. While when an instance with a smaller instance id got swapped out, its original segments will be assigned to the next available instance with a slightly larger instance id. In this case, all the segments will be moved around.
If we can retain the existing sequence of the instances and only replace the missing one, we can minimize the segment movement. This will be helpful for those large Pinot clusters to minimize the segment movement and avoid suddenly heavy read/write access to the deep storage.
Contributor guide
Research direction
The issue does not name files, tests, or an entry point. Start by tracing the table rebalance request and locating where instances are sorted before partitioning. Done means preserving the existing instance sequence, replacing only missing instances, and minimizing unnecessary segment movement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100