Modify the findMerges policy for addIndexes(CodecReader[]) to create 1 segment per reader [LUCENE-10476]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Once we have #11253, we can trigger concurrent merges for addIndexes(CodecReader...) api, These merges are defined in the MergePolicy and triggered by the MergeScheduler.
We can now create a merge policy that does a 1:1 addIndexes call, where the merger creates one segment each for every provided reader. It would create a faster, more concurrent `addIndexes(CodecReader...)`}, at the cost of deferring some merges to be done later in background.
Which, I believe is similar to the behavior in `addIndexes(Directory...)` - all incoming segments are simply added to IW, and any merging happens in later in background.
The change, would make both APIs behave consistently.
---
Migrated from [LUCENE-10476](https://issues.apache.org/jira/browse/LUCENE-10476) by Vigya Sharma (@vigyasharma)
Parent: #11253
Contributor guide
Research direction
Start with parent issue #11253 and the addIndexes(CodecReader...) and addIndexes(Directory...) entry points, then trace how MergePolicy and MergeScheduler handle the incoming readers. The change is done when CodecReader additions create one segment per reader, matching Directory behavior while deferring subsequent merges to background processing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100