apache / apache/lucene

FSTs are Always Run in OFF_HEAP Load Mode if Directory not MMAPed

Open
#11,730 1 comment 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

In testing internally, we have found that FSTs always load in OFF_HEAP mode. If you are not using an MMAP directory, some queries could result in system call overhead via NIO FS. While there are obviously use cases that are not impacted by the load mode of FSTs and may have a use for NIO FS, we don't use the `ON_HEAP` load mode from anywhere I or my colleagues could see in Lucene.

In turn, does it make sense to remove that load mode going forward?

Contributor guide

Open the contributing guide

Research direction

Start by tracing where FST load modes are selected for MMAP and non-MMAP directories, focusing on the reported OFF_HEAP behavior and NIO FS overhead. Review existing coverage for FST loading and query behavior, then establish whether removing ON_HEAP is safe across the affected directory use cases before defining the required changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.