apache / apache/lucene

Can Lucene support selective preloading of just slices/files within a CFS file?

Open
#13,967 2 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

At Amazon Product Search, we warm up our service by preloading vector files into RAM and use the `mmapDir.setPreload` API for this.
However, when vector files get compacted into a `.cfs` file, they do not get paged into RAM due to `setPreload` being called on the basis of `.vec`, `.veq` extensions.

This would also help for lexical search files that get compacted into `.cfs` files and which need preloading for service warm up purposes..

Contributor guide

Open the contributing guide

Research direction

Start by reading the mmapDir.setPreload API and the CFS behavior described in the issue; trace how preload selection currently matches .vec and .veq files. Done means callers can selectively preload slices or files inside a .cfs file, covering both vector and lexical search files, with regression tests for the supported selection behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.