apache / apache/pinot

Support balanced segment assignment for consuming segments

Open
#8,585 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently for consuming segments, the following formula is used for segment assignment:
```
(partitionId * numReplicas + replicaId) % numInstances
```
This gives us unbalanced assignment which is shown in the following examples:
```
Scenario#1:

10 instances -> i0, i1, ..., i9
2 partitions -> 0, 1
3 replicas -> 0, 1, 2

segment name: partitionId_seqId(replicaId)

i0 i1 i2 i3 i4 i5 i6 i7 i8 i9
0_0(0) 0_0(1) 0_0(2) 1_0(0) 1_0(1) 1_0(2)
0_1(0) 0_1(1) 0_1(2) 1_1(0) 1_1(1) 1_1(2)
0_2(0) 0_2(1) 0_2(2) 1_2(0) 1_2(1) 1_2(2)
0_3(0) 0_3(1) 0_3(2) 1_3(0) 1_3(1) 1_3(2)
0_4(0) 0_4(1) 0_4(2) 1_4(0) 1_4(1) 1_4(2)
0_5(0) 0_5(1) 0_5(2) 1_5(0) 1_5(1) 1_5(2)
0_6(0) 0_6(1) 0_6(2) 1_6(0) 1_6(1) 1_6(2)

Scenario#2:

5 instances
2 partitions
3 replicas

i0 i1 i2 i3 i4
0_0(0) 0_0(1) 0_0(2) 1_0(0) 1_0(1)
1_0(2)
0_1(0) 0_1(1) 0_1(2) 1_1(0) 1_1(1)
1_1(2)
0_2(0) 0_2(1) 0_2(2) 1_2(0) 1_2(1)
1_2(2)
0_3(0) 0_3(1) 0_3(2) 1_3(0) 1_3(1)
1_3(2)
0_4(0) 0_4(1) 0_4(2) 1_4(0) 1_4(1)
1_4(2)
0_5(0) 0_5(1) 0_5(2) 1_5(0) 1_5(1)
1_5(2)
0_6(0) 0_6(1) 0_6(2) 1_6(0) 1_6(1)
1_6(2)
```
In scenario#1, 4 instances don't get any segments assigned to them.
In scenario#2, instance `i0` has two times more segments than the other instances.

The existing realtime segment assignment tries to put all segments of the same partition in one instance. That's good for the use cases for which the issued queries always have partitioned column in the where-clause so that query execution will be faster.
But that's not a hard requirement for all realtime tables. Having a balanced segment assignment leads to a balanced load distribution on the servers for the use case that don't have that requirement.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing realtime segment assignment described in the issue, then reproduce both scenarios with their instance, partition, and replica counts. Done means consuming segments are distributed across instances without the demonstrated imbalance while retaining the existing partition-locality behavior where applicable.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.