UpgradeIndexMergePolicy: beyond one-off use, monster segment avoidance [LUCENE-7523]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
(Was looking at UpgradeIndexMergePolicy as part of [SOLR-9648](https://issues.apache.org/jira/browse/SOLR-9648) and came up with these possibilities here, what do people think?)
Currently one probably would not configure use of the `UpgradeIndexMergePolicy` (UIMP) permanently since [findForcedMerges](https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/UpgradeIndexMergePolicy.java#L74) becomes a no-op after all segments have been upgraded.
- How about adding an optional `fallbackToInnerAfterUpgrade` flag? That way UIMP.findForcedMerges could fallback to its inner/delegate merge policy's findForcedMerges call after all segments have been upgraded.
Currently UIMP.findForcedMerges identifies all the segments to be upgraded and then asks its inner/delegate merge policy to come up with a MergeSpecification for those segments. If the inner/delegate merge policy does not supply a merge for all the segments to be upgraded then UIMP merges the remaining segments into _one_ new segment. That extra new segment could be quite a large 'monster' segment.
- How about adding an optional `upgradeUnmergedSegmentsIndividually` flag? That way UIMP.findForcedMerges could upgrade (but not merge) the remaining segments.
- Or indeed should 'upgradeUnmergedSegmentsIndividually' be the default behaviour?
Noticed whilst looking at the code:
- [UIMP.findMerges](https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/UpgradeIndexMergePolicy.java#L69) does not pass the mergeTrigger to the inner/delegate merge policy.
- If we can figure out why that is, let's add a comment to say why that is.
- Understanding why that is would also be needed before proceeding with beyond one-off use of UIMP.
---
Migrated from [LUCENE-7523](https://issues.apache.org/jira/browse/LUCENE-7523) by Christine Poerschke (@cpoerschke), updated Apr 22 2018
Attachments: [LUCENE-7523-outline.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7523/LUCENE-7523-outline.patch)
Linked issues:
- [SOLR-9648](https://issues.apache.org/jira/browse/SOLR-9648)
- [SOLR-12259](https://issues.apache.org/jira/browse/SOLR-12259)
Contributor guide
Research direction
Start with lucene/core/src/java/org/apache/lucene/index/UpgradeIndexMergePolicy.java, especially findMerges and findForcedMerges, and review the linked LUCENE-7523-outline.patch. Resolve the proposed fallback and individual-upgrade behavior, including why findMerges does not pass mergeTrigger; done requires an agreed scope and corresponding implementation coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100