Index upgrader should have option to do multiple segments instead of one [LUCENE-5976]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Right now, index upgrader produces one gigantic segment. This can take a long time, consume more memory than normal merges, and even sidestep the max seg size for the delegated MergePolicy.
It would be nice to have a simpler option: create 1 upgraded segment for every existing segment. If there are deletes that are merged away, the regulard MP can takeover after the upgrade is complete (or even partially complete) to merge away deletes.
---
Migrated from [LUCENE-5976](https://issues.apache.org/jira/browse/LUCENE-5976) by Ryan Ernst (@rjernst)
Contributor guide
Research direction
No source file or test is named in the issue. Start by locating the index upgrader and its merge-policy integration, then determine how an option could produce one upgraded segment per existing segment. Done means upgrades no longer require one gigantic segment and the regular merge policy can handle remaining deletes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100