apache / apache/pinot

Broker routing time is impact by the time pruner with 10k+ segments

Open
#8,931 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

We seen for tables where we have 10 thousand or more partitions, broker routing time will increase to 100ms or more. We have not had a chance to profile yet, but the theory is that serializing/deserializing interval trees with large numbers of segments is causing issues.

Contributor guide

Open the contributing guide

Research direction

Start by profiling broker routing for tables with 10,000 or more partitions and inspect the time pruner's interval-tree serialization and deserialization path. Confirm whether those operations account for the reported 100ms or higher routing time, then validate the result against a large-segment case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.