apache / apache/lucene

Dynamic Index Cache Sizing [LUCENE-10602]

Open
#11,638 9 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Working with Lucene's filter cache, it has become apparent that it can be an enormous drain on the heap and therefore the JVM. After extensive usage of an index, it is not uncommon to tune performance by shrinking or altogether removing the filter cache.

Lucene tracks hit/miss stats of the filter cache, but it does nothing with the data other than inform an interested user about the effectiveness of their index's caching.

It would be interesting if Lucene would be able to tune the index filter cache heuristically based on actual usage (age, frequency, and value).

This could ultimately be used to give GBs of heap back to an individual Lucene instance instead of burning it on cache storage that's not effectively used (or useful).

---
Migrated from [LUCENE-10602](https://issues.apache.org/jira/browse/LUCENE-10602) by Chris Earle (@pickypg), updated Jun 09 2022

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start by locating Lucene's filter cache implementation and the existing hit/miss statistics, then review how cache age, frequency, and value could be evaluated. The issue is not done until a specific heuristic, behavior, and corresponding tests are agreed upon.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.