DruidSchema and SystemSchema should avoid creating temporary collections of the cardinality of all segments
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 31m
- Merged PRs (30d)
- 209
Description
It happens in `DruidSchema.getSegmentMetadata()` and `SystemSchema.scan()`. Creating such collections means allocating gigabytes of data, if there are millions of segments in the system. It should be done only if there is really no other way. Avoiding allocations in `DruidSchema.getSegmentMetadata()` and `SystemSchema.scan()` is not very easy without introducing concurrency bugs, but I believe should be possible.
Contributor guide
Research direction
Start by reading DruidSchema.getSegmentMetadata() and SystemSchema.scan(), tracing where temporary collections are created and how their data is consumed. Compare their concurrency assumptions before choosing an allocation-safe approach. Done means avoiding collections proportional to all segments without introducing concurrency bugs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100