apache / apache/pinot

Retain instance sequence for table rebalance

Open
#8,371 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently when the table rebalance request is made, the list of instances will be sorted by the instance ids before the instance partition is formed. While when an instance with a smaller instance id got swapped out, its original segments will be assigned to the next available instance with a slightly larger instance id. In this case, all the segments will be moved around.
If we can retain the existing sequence of the instances and only replace the missing one, we can minimize the segment movement. This will be helpful for those large Pinot clusters to minimize the segment movement and avoid suddenly heavy read/write access to the deep storage.

Contributor guide

Open the contributing guide

Research direction

The issue does not name files, tests, or an entry point. Start by tracing the table rebalance request and locating where instances are sorted before partitioning. Done means preserving the existing instance sequence, replacing only missing instances, and minimizing unnecessary segment movement.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.