apache / apache/lucene

UpgradeIndexMergePolicy: beyond one-off use, monster segment avoidance [LUCENE-7523]

Open
#8,574 2 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

(Was looking at UpgradeIndexMergePolicy as part of [SOLR-9648](https://issues.apache.org/jira/browse/SOLR-9648) and came up with these possibilities here, what do people think?)

Currently one probably would not configure use of the `UpgradeIndexMergePolicy` (UIMP) permanently since [findForcedMerges](https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/UpgradeIndexMergePolicy.java#L74) becomes a no-op after all segments have been upgraded.
- How about adding an optional `fallbackToInnerAfterUpgrade` flag? That way UIMP.findForcedMerges could fallback to its inner/delegate merge policy's findForcedMerges call after all segments have been upgraded.

Currently UIMP.findForcedMerges identifies all the segments to be upgraded and then asks its inner/delegate merge policy to come up with a MergeSpecification for those segments. If the inner/delegate merge policy does not supply a merge for all the segments to be upgraded then UIMP merges the remaining segments into _one_ new segment. That extra new segment could be quite a large 'monster' segment.
- How about adding an optional `upgradeUnmergedSegmentsIndividually` flag? That way UIMP.findForcedMerges could upgrade (but not merge) the remaining segments.
- Or indeed should 'upgradeUnmergedSegmentsIndividually' be the default behaviour?

Noticed whilst looking at the code:
- [UIMP.findMerges](https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/index/UpgradeIndexMergePolicy.java#L69) does not pass the mergeTrigger to the inner/delegate merge policy.
- If we can figure out why that is, let's add a comment to say why that is.
- Understanding why that is would also be needed before proceeding with beyond one-off use of UIMP.

---
Migrated from [LUCENE-7523](https://issues.apache.org/jira/browse/LUCENE-7523) by Christine Poerschke (@cpoerschke), updated Apr 22 2018
Attachments: [LUCENE-7523-outline.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7523/LUCENE-7523-outline.patch)
Linked issues:
- [SOLR-9648](https://issues.apache.org/jira/browse/SOLR-9648)
- [SOLR-12259](https://issues.apache.org/jira/browse/SOLR-12259)

Contributor guide

Open the contributing guide

Research direction

Start with lucene/core/src/java/org/apache/lucene/index/UpgradeIndexMergePolicy.java, especially findMerges and findForcedMerges, and review the linked LUCENE-7523-outline.patch. Resolve the proposed fallback and individual-upgrade behavior, including why findMerges does not pass mergeTrigger; done requires an agreed scope and corresponding implementation coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.