apache / apache/druid

Generate interleavings in MetadataSegmentManager.markAsUnusedAllSegmentsInDataSource()

Open
#8,169 0 comments 0 reactions 0 assignees View on GitHub
Area - Metadata Contributions Welcome Performance
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

If `MetadataSegmentManager.markAsUnusedAllSegmentsInDataSource()` is called for a large data source, it may hang up the SQL table for some time. The code may generate interleavings and update only, say, 1000 entries at a time, and sleep between these "batches" for several seconds.

Contributor guide

Open the contributing guide

Research direction

Start at MetadataSegmentManager.markAsUnusedAllSegmentsInDataSource() and inspect how it currently updates segments for a large data source. Determine how batching, interleaving, and pauses should fit the existing operation. Done means large-data-source calls update bounded batches with pauses instead of holding up the SQL table for an extended period.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.