apache / apache/lucene

Performance regression in NRTCachingDirectory [LUCENE-9972]

Open
#11,011 5 comments 0 reactions 0 assignees View on GitHub
affects-version:8.8.2 legacy-jira-label:performance legacy-jira-priority:Major module:core/store type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

This issue has been discussed here:

 

Summary: To get the same speed as in 8.3 I have to overwrite NRTCachingDirectory#doCacheWrite

in 8.8 with the implementation from 8.3. Otherwise it's magnitudes slower.

 

It seems that the solution for #10156 increases the number of calls to native-OpenFile (due to falling down to MMAP-Directory) by a very  large amount.

I understand the reasoning behind #10156 but in fact it destroys the performance of our product. For now sub-classing NRTCachingDirectory with the old implementation of doCacheWrite() solves the issue but it would be great to have an "offical" solution of course.

---
Migrated from [LUCENE-9972](https://issues.apache.org/jira/browse/LUCENE-9972) by Markus Gietzen, updated Aug 24 2021

Contributor guide

Open the contributing guide

Research direction

Start with NRTCachingDirectory#doCacheWrite and compare the 8.8 behavior with the 8.3 implementation referenced in the issue. Read the discussion around #10156 and the linked mailing-list thread, then investigate the increased native-OpenFile calls from falling back to MMAP-Directory. Done means an official fix restores the reported performance without requiring a subclass override.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.