apache / apache/druid

DruidSchema and SystemSchema should avoid creating temporary collections of the cardinality of all segments

Open
#6,827 5 comments 0 reactions 0 assignees View on GitHub
Area - SQL Performance
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 31m
Merged PRs (30d)
209

Description

It happens in `DruidSchema.getSegmentMetadata()` and `SystemSchema.scan()`. Creating such collections means allocating gigabytes of data, if there are millions of segments in the system. It should be done only if there is really no other way. Avoiding allocations in `DruidSchema.getSegmentMetadata()` and `SystemSchema.scan()` is not very easy without introducing concurrency bugs, but I believe should be possible.

Contributor guide

Open the contributing guide

Research direction

Start by reading DruidSchema.getSegmentMetadata() and SystemSchema.scan(), tracing where temporary collections are created and how their data is consumed. Compare their concurrency assumptions before choosing an allocation-safe approach. Done means avoiding collections proportional to all segments without introducing concurrency bugs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.