apache / apache/lucene

Should we support preloading slices of MemorySegmentIndexInput?

Open
#15,579 1 comment 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

The [`Lucene90CompoundFormat`](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90CompoundFormat.java) combines all files in a segment into a single CFS file. At the time of reading, it [opens the single combined file once](https://github.com/apache/lucene/blob/cf41f22142d3c8360a4d4f380a391e737eccfa88/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90CompoundReader.java#L76) using `Directory#openInput` and subsequent files are ["opened" by simply slicing that input](https://github.com/apache/lucene/blob/cf41f22142d3c8360a4d4f380a391e737eccfa88/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90CompoundReader.java#L149).

For this format, the ability to [preload files from `MMapDirectory`](https://github.com/apache/lucene/blob/cf41f22142d3c8360a4d4f380a391e737eccfa88/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L387-L388) is binary -- we either preload the entire index (i.e. CFS file), or none of it.. (because preload happens per-file, in `MMapDirectory#openInput`)

I wonder if we can move preloading functionality down from `MMapDirectory` to `MemorySegmentIndexInput` ([like madvise calls](https://github.com/apache/lucene/blob/cf41f22142d3c8360a4d4f380a391e737eccfa88/lucene/core/src/java/org/apache/lucene/store/MemorySegmentIndexInput.java#L577-L601)), allowing us to load chunks of a larger file based on `IOContext` -> hints -> [`PreloadHint`](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/store/PreloadHint.java).

Contributor guide

Open the contributing guide

Research direction

Start with Lucene90CompoundReader.java and MMapDirectory.java to trace how a compound file is opened and sliced, then read MemorySegmentIndexInput.java around the madvise methods and PreloadHint.java. Determine whether preload hints can be applied to slices without breaking existing behavior; done would require a decided approach and corresponding validation for compound-file reads.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.