Broker routing time is impact by the time pruner with 10k+ segments
Open
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
We seen for tables where we have 10 thousand or more partitions, broker routing time will increase to 100ms or more. We have not had a chance to profile yet, but the theory is that serializing/deserializing interval trees with large numbers of segments is causing issues.
Contributor guide
Research direction
Start by profiling broker routing for tables with 10,000 or more partitions and inspect the time pruner's interval-tree serialization and deserialization path. Confirm whether those operations account for the reported 100ms or higher routing time, then validate the result against a large-segment case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100