apache / apache/lucene

Modify the findMerges policy for addIndexes(CodecReader[]) to create 1 segment per reader [LUCENE-10476]

Open
#11,512 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Once we have #11253, we can trigger concurrent merges for addIndexes(CodecReader...) api, These merges are defined in the MergePolicy and triggered by the MergeScheduler.

We can now create a merge policy that does a 1:1 addIndexes call, where the merger creates one segment each for every provided reader. It would create a faster, more concurrent `addIndexes(CodecReader...)`}, at the cost of deferring some merges to be done later in background.

Which, I believe is similar to the behavior in `addIndexes(Directory...)` - all incoming segments are simply added to IW, and any merging happens in later in background.

The change, would make both APIs behave consistently.

---
Migrated from [LUCENE-10476](https://issues.apache.org/jira/browse/LUCENE-10476) by Vigya Sharma (@vigyasharma)
Parent: #11253

Contributor guide

Open the contributing guide

Research direction

Start with parent issue #11253 and the addIndexes(CodecReader...) and addIndexes(Directory...) entry points, then trace how MergePolicy and MergeScheduler handle the incoming readers. The change is done when CodecReader additions create one segment per reader, matching Directory behavior while deferring subsequent merges to background processing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.