apache / apache/lucene

Consider implementing "lookback" merge policy [LUCENE-5134]

Open
#6,198 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

In http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html Mike mentioned lookahead as something that could possibly yield more
optimal merges.

But what about lookback?

What if some sort of stats were kept about about which segments were
picked for merges? With some sort of stats in hand, could one look
back and, knowing what happened after those merges, evaluate if more
optimal merge choices could have been made and then use that "next
time"?

See http://search-lucene.com/m/D7ypz1gT2H91

---
Migrated from [LUCENE-5134](https://issues.apache.org/jira/browse/LUCENE-5134) by Otis Gospodnetic (@otisg)

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by reading the linked merge discussion and locating Lucene’s segment merge policy implementation; then define the required statistics, feedback loop, and evaluation criteria before assessing feasibility. Done would require an agreed design and tests demonstrating whether lookback produces better merge choices.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.