SqlMetadataSegmentManager should use batching for querying and updating segments when possible
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
For example, in this method: https://github.com/apache/incubator-druid/blob/c648775b5b9001dafd26192abf2a61f44bfc16d4/server/src/main/java/org/apache/druid/metadata/SQLMetadataSegmentManager.java#L262-L284 should partition segments may be in blocks of 100 segments and use IN operator to query segments from the DB rather than issuing a query for every segment in separation.
Same for https://github.com/apache/incubator-druid/blob/c648775b5b9001dafd26192abf2a61f44bfc16d4/server/src/main/java/org/apache/druid/metadata/SQLMetadataSegmentManager.java#L386-L414
FYI @dampcake @surekhasaharan
Contributor guide
Research direction
Start in server/src/main/java/org/apache/druid/metadata/SQLMetadataSegmentManager.java at lines 262-284 and 386-414. Read how those methods query and update segments, then verify the database access patterns and existing tests before changing them. Done means both paths batch segments in groups of about 100 and use IN queries instead of one query per segment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- database
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100