apache / apache/lucene

Hot, cold or adaptive index mode selection

Open
#15,515 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

In https://github.com/apache/lucene/pull/15458 we found that cold index optimizations (deferring term states to send the madvise WILLNEED signal) can hurt performance when running a hot index. This might also be true for other cases where Lucene uses madvise?

We could have a parameter for users to select hot or cold index mode, depending on their use case.

We can also try to implement an adaptive mode, which enables cold index optimizations when the index is cold and enables hot index optimizations otherwise. I thought (https://github.com/apache/lucene/pull/15458#issuecomment-3617935050) we could use the `isLoaded` method for it, but as @mikemccand pointed out, this method is quite expensive (https://github.com/apache/lucene/pull/14156#issuecomment-3649593085). I think we can implement an `isProbablyLoaded` method (which checks only the first, middle, and last pages of a memory segment) to overcome that.

As for `TermStates.get` itself, we can check if deferring dictionary lookup improves cold index performance (https://github.com/apache/lucene/pull/15458#issuecomment-3618633775 ). If it does, I think we can propagate the `isProbablyLoaded` method results to it and defer the dictionary lookup only if the index is cold.

We can also look at other madvise use cases and see if we can apply the same mechanism to them?

Contributor guide

Open the contributing guide

Research direction

Start by reviewing pull request 15458 and its linked comments, then inspect TermStates.get, isLoaded, and the other madvise use cases mentioned in the issue. Establish whether hot, cold, or adaptive selection is wanted and how it should be evaluated; done requires an agreed design supported by performance measurements.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.