apache / apache/pinot

Performance improvement needed when pushing segments for table with lots of segments

Open
#6,275 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

Right now if we have 100k+ segments, pushing new segments into the table is very slow.

**In general, we need to find a strategy to make pushing lots of segments faster**

Below are some ideas for tracking:

One possible direct reason is that the update on IdealState of segments will be very slow.

By checking jstack snapshots, I can see that when we update IdealState, we try to clone the IdealState and then compare the new value with old value.

This is fine for small IdealStates, but currently the segment assignment states are being cloned by "Serializing => (Json) => Deserializing" which maybe is the reason why it is slow.

Another idea is maybe we could have a segment push queue, which merges the segment push results and do a batch update on ideal states.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or concrete entry points are named. Start by profiling the IdealState update path with jstack snapshots, then compare the serialization-based clone and comparison cost with the proposed segment-push queue and batching approach; done requires a demonstrated strategy that makes pushes for tables with 100k+ segments faster.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.