apache / apache/druid

SqlMetadataSegmentManager should use batching for querying and updating segments when possible

Open
#7,568 2 comments 0 reactions 0 assignees View on GitHub
Performance
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

For example, in this method: https://github.com/apache/incubator-druid/blob/c648775b5b9001dafd26192abf2a61f44bfc16d4/server/src/main/java/org/apache/druid/metadata/SQLMetadataSegmentManager.java#L262-L284 should partition segments may be in blocks of 100 segments and use IN operator to query segments from the DB rather than issuing a query for every segment in separation.

Same for https://github.com/apache/incubator-druid/blob/c648775b5b9001dafd26192abf2a61f44bfc16d4/server/src/main/java/org/apache/druid/metadata/SQLMetadataSegmentManager.java#L386-L414

FYI @dampcake @surekhasaharan

Contributor guide

Open the contributing guide

Research direction

Start in server/src/main/java/org/apache/druid/metadata/SQLMetadataSegmentManager.java at lines 262-284 and 386-414. Read how those methods query and update segments, then verify the database access patterns and existing tests before changing them. Done means both paths batch segments in groups of about 100 and use IN queries instead of one query per segment.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
database
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.