Performance improvement needed when pushing segments for table with lots of segments
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
Right now if we have 100k+ segments, pushing new segments into the table is very slow.
**In general, we need to find a strategy to make pushing lots of segments faster**
Below are some ideas for tracking:
One possible direct reason is that the update on IdealState of segments will be very slow.
By checking jstack snapshots, I can see that when we update IdealState, we try to clone the IdealState and then compare the new value with old value.
This is fine for small IdealStates, but currently the segment assignment states are being cloned by "Serializing => (Json) => Deserializing" which maybe is the reason why it is slow.
Another idea is maybe we could have a segment push queue, which merges the segment push results and do a batch update on ideal states.
Contributor guide
Research direction
No files, tests, or concrete entry points are named. Start by profiling the IdealState update path with jstack snapshots, then compare the serialization-based clone and comparison cost with the proposed segment-push queue and batching approach; done requires a demonstrated strategy that makes pushes for tables with 100k+ segments faster.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100