incorrect results from SortedNumericDocValuesRangeQuery when using double/float based points+NumericDocValuesField
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
When attempting to upgrade Solr from Lucene 10.4.0 -> 10.5.1 the Solr tests identified a regression causing test failures in (single-valued) float & double based fields that use both points & docvalues: Range queries are failing to match all expected documents.
* Identical Solr tests using multivalued fields still pass.
* Identical Solr tests using _only_ docvalues, or _only_ points also still pass.
* Similar Solr tests using int & long types still pass regardless of multivalue/points/dv permutations
After writing a scaled down Lucene equivilent test, git bisect identified `da18ffc5890f72df279373d40ce5c4192734cd11` (aka: #15760) as the cause of the bug.
I'm not an expert here, but the underlying issue seems to be that #15760 modified `SortedNumericDocValuesRangeQuery.rewrite` to refactor some logic identifying min/max values "on disk" (in order to optimize away queries known to be out of range of any valid docs) into a new helper function (`NumericFieldStats.getStats`) that not only considers the stats of the docvalues (if they exist), but (starting with #15760) this method also (first) looks at the stats from any `PointValues` -- and those stats may differ from what's _actually_ in the doc values.
The problem being that there is no guarantee that a `PointsField` named `foo` has the same internal `long` values in it as a `DocValues` field named `foo`.
IIUC, fields like `DoublePoint` and `FloatPoint` internally use the same encodings methods in `NumericUtils` that are recommended when using `SortedNumericDocValuesField` -- but AFAIK there is no requirement that all doubles & floats be encoded that way when using `NumericDocValuesField` -- which is also supported by `SortedNumericDocValuesRangeQuery`.
* Solr uses `Double.doubleToLongBits` with `NumericDocValuesField` for historic reasons (predating Lucene's `NumericUtils` IIRC?)
* likewise for Floats
* Lucene's `DoubleDocValuesField` helper class uses `Double.doubleToRawLongBits`
* likewise for Floats
* Neither seem to safely work with this new `NumericFieldStats.getStats()` logic
I'm attaching a quickly hacked together patch with a test case demonstrating the problem:
* [lucene_bug_caused_by_github15760.patch](https://github.com/user-attachments/files/31533966/lucene_bug_caused_by_github15760.patch). This tests passes with Lucene 10.4.0
I suspect the only viable fix here is to remove the `PointValues` optimizations added in #15760 ... or ... _maybe_: make the `PointValues` optimization used if-and-only-if we can confirm from the IndexReader that `SortedNumericDocValues` are in use -- since `SortedNumericDocValues` they are documented as requiring the use of the sae `NumericUtils` encoding for double & float that Points use (IIUC this is easy/cheap to do? ... just check if `null != LeafReader.getSortedNumericDocValues()` ... correct?)
### Version and environment details
Lucene 10.5.1
Contributor guide
Research direction
Start with the attached reduced test and trace SortedNumericDocValuesRangeQuery.rewrite into NumericFieldStats.getStats. Compare the PointValues and SortedNumericDocValues paths, including LeafReader.getSortedNumericDocValues(), for single-valued float and double fields using both points and doc values. Done means the regression test passes and the existing multivalued, points-only, doc-values-only, int, and long cases remain correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100