Can Lucene support selective preloading of just slices/files within a CFS file?
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
At Amazon Product Search, we warm up our service by preloading vector files into RAM and use the `mmapDir.setPreload` API for this.
However, when vector files get compacted into a `.cfs` file, they do not get paged into RAM due to `setPreload` being called on the basis of `.vec`, `.veq` extensions.
This would also help for lexical search files that get compacted into `.cfs` files and which need preloading for service warm up purposes..
Contributor guide
Research direction
Start by reading the mmapDir.setPreload API and the CFS behavior described in the issue; trace how preload selection currently matches .vec and .veq files. Done means callers can selectively preload slices or files inside a .cfs file, covering both vector and lexical search files, with regression tests for the supported selection behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100