Performance regression in NRTCachingDirectory [LUCENE-9972]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
This issue has been discussed here:
Summary: To get the same speed as in 8.3 I have to overwrite NRTCachingDirectory#doCacheWrite
in 8.8 with the implementation from 8.3. Otherwise it's magnitudes slower.
It seems that the solution for #10156 increases the number of calls to native-OpenFile (due to falling down to MMAP-Directory) by a very large amount.
I understand the reasoning behind #10156 but in fact it destroys the performance of our product. For now sub-classing NRTCachingDirectory with the old implementation of doCacheWrite() solves the issue but it would be great to have an "offical" solution of course.
---
Migrated from [LUCENE-9972](https://issues.apache.org/jira/browse/LUCENE-9972) by Markus Gietzen, updated Aug 24 2021
Contributor guide
Research direction
Start with NRTCachingDirectory#doCacheWrite and compare the 8.8 behavior with the 8.3 implementation referenced in the issue. Read the discussion around #10156 and the linked mailing-list thread, then investigate the increased native-OpenFile calls from falling back to MMAP-Directory. Done means an official fix restores the reported performance without requiring a subclass override.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100