apache / apache/lucene

UninvertingReader cannot be used with ControlledRealTimeReopenThread [LUCENE-6370]

Open
#7,430 6 comments 0 reactions 0 assignees View on GitHub
affects-version:5.0 legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

In order to sort over non-DocValues fields in 5.0 we need to use an UninvertingReader to get the old FieldCache behavior back. However, UninvertingReader cannot be (easily) used with a ControlledRealTimeReopenThread.

Specifically, the easiest way to construct a ControlledRealTimeReopenThread is with a SearcherManager. The only way I found to wire an UninvertingReader into SearcherManager is to implement a SearcherFactory that wraps the passed-in reader. Unfortunately, that runs afoul of the check in SearcherManager.getSearcher that requires "SearcherFactory must wrap exactly the provided reader". So, as long as this check is there, I simply don't see a way to use UninvertingReader with NRT functionality.

I think this is a serious issue for programs that need to be able to use NRT search features on indexes created with previous Lucene versions, for whom upgrading the index is not an easy option. If they were previously relying on sorting implicitly via FieldCache, the _only_ ways to upgrade are:
a) rebuild the index using DocValues fields, or
b) use UninvertingReader
Right now there's a catch-22, as (a) is assumed to be not an option and (b) is broken due to this bug.

I have a hacky workaround for Gerrit Code Review up for review here:
https://gerrit-review.googlesource.com/#/c/66613/6/gerrit-lucene/src/main/java/com/google/gerrit/lucene/WrappableSearcherManager.java@191
Basically, it loosens the restriction on the newSearcher result to allow Filtered{Directory,Leaf}Readers that wrap the original reader. This appears to work fine for us, and I don't see anything in UninvertingReader that would cause me to believe it doesn't work. However, I'm no expert on Lucene internals and I don't know why that identity check was there in the first place, so I may be missing something.

Please do not take that patch directly until I have gotten permission from my employer to contribute it.

---
Migrated from [LUCENE-6370](https://issues.apache.org/jira/browse/LUCENE-6370) by Dave Borowitz, updated Mar 26 2015
Attachments: [LUCENE-6370.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-6370/LUCENE-6370.patch)

Contributor guide

Open the contributing guide

Research direction

Start at SearcherManager.getSearcher and the SearcherFactory path, then read how ControlledRealTimeReopenThread obtains its manager. Review the referenced LUCENE-6370.patch and UninvertingReader wrapping behavior; done means establishing a safe way to combine the wrapper with NRT search without violating the reader validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.