Should we support preloading slices of MemorySegmentIndexInput?
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
The [`Lucene90CompoundFormat`](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90CompoundFormat.java) combines all files in a segment into a single CFS file. At the time of reading, it [opens the single combined file once](https://github.com/apache/lucene/blob/cf41f22142d3c8360a4d4f380a391e737eccfa88/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90CompoundReader.java#L76) using `Directory#openInput` and subsequent files are ["opened" by simply slicing that input](https://github.com/apache/lucene/blob/cf41f22142d3c8360a4d4f380a391e737eccfa88/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90CompoundReader.java#L149).
For this format, the ability to [preload files from `MMapDirectory`](https://github.com/apache/lucene/blob/cf41f22142d3c8360a4d4f380a391e737eccfa88/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L387-L388) is binary -- we either preload the entire index (i.e. CFS file), or none of it.. (because preload happens per-file, in `MMapDirectory#openInput`)
I wonder if we can move preloading functionality down from `MMapDirectory` to `MemorySegmentIndexInput` ([like madvise calls](https://github.com/apache/lucene/blob/cf41f22142d3c8360a4d4f380a391e737eccfa88/lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java#L577-L601)), allowing us to load chunks of a larger file based on `IOContext` -> hints -> [`PreloadHint`](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/store/PreloadHint.java).
Contributor guide
Research direction
Start with Lucene90CompoundReader.java and MMapDirectory.java to trace how a compound file is opened and sliced, then read MemorySegmentIndexInput.java around the madvise methods and PreloadHint.java. Determine whether preload hints can be applied to slices without breaking existing behavior; done would require a decided approach and corresponding validation for compound-file reads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100