Generate interleavings in MetadataSegmentManager.markAsUnusedAllSegmentsInDataSource()
Open
Area - Metadata
Contributions Welcome
Performance
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
If `MetadataSegmentManager.markAsUnusedAllSegmentsInDataSource()` is called for a large data source, it may hang up the SQL table for some time. The code may generate interleavings and update only, say, 1000 entries at a time, and sleep between these "batches" for several seconds.
Contributor guide
Research direction
Start at MetadataSegmentManager.markAsUnusedAllSegmentsInDataSource() and inspect how it currently updates segments for a large data source. Determine how batching, interleaving, and pauses should fit the existing operation. Done means large-data-source calls update bounded batches with pauses instead of holding up the SQL table for an extended period.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100