Hot, cold or adaptive index mode selection
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
In https://github.com/apache/lucene/pull/15458 we found that cold index optimizations (deferring term states to send the madvise WILLNEED signal) can hurt performance when running a hot index. This might also be true for other cases where Lucene uses madvise?
We could have a parameter for users to select hot or cold index mode, depending on their use case.
We can also try to implement an adaptive mode, which enables cold index optimizations when the index is cold and enables hot index optimizations otherwise. I thought (https://github.com/apache/lucene/pull/15458#issuecomment-3617935050) we could use the `isLoaded` method for it, but as @mikemccand pointed out, this method is quite expensive (https://github.com/apache/lucene/pull/14156#issuecomment-3649593085). I think we can implement an `isProbablyLoaded` method (which checks only the first, middle, and last pages of a memory segment) to overcome that.
As for `TermStates.get` itself, we can check if deferring dictionary lookup improves cold index performance (https://github.com/apache/lucene/pull/15458#issuecomment-3618633775 ). If it does, I think we can propagate the `isProbablyLoaded` method results to it and defer the dictionary lookup only if the index is cold.
We can also look at other madvise use cases and see if we can apply the same mechanism to them?
Contributor guide
Research direction
Start by reviewing pull request 15458 and its linked comments, then inspect TermStates.get, isLoaded, and the other madvise use cases mentioned in the issue. Establish whether hot, cold, or adaptive selection is wanted and how it should be evaluated; done requires an agreed design supported by performance measurements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100