apache / apache/lucene

Lucene104ScalarQuantizedVectorFormat returns different scores after merging

Open
#15,408 15 comments 0 reactions 1 assignee Claimed by @mccullocht View on GitHub
type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

We've found a bug in `Lucene104ScalarQuantizedVectorFormat` where the scorer is returning different documents scores after merging segments together. The test case can be found [here](https://github.com/thecoop/lucene/blob/scalar-quantizer-merge-bug/lucene/core/src/test/org/apache/lucene/codecs/lucene104/TestLucene104ScalarQuantizedVectorsFormat.java#L79) (on `branch_10x`)

Before merge, docs returned and scores are:

`{0=3.5892557207262143E-5, 1=6.082725012674928E-4, 2=1.0}`

After merge, docs returned and scores are:

`{0=3.581063720048405E-5, 1=6.144083454273641E-4, 2=0.0}`

Doc 2 (which matches the query vector), has changed from 1.0 to 0.0

This only occurs with the `PACKED_NIBBLE` encoding.

### Version and environment details

Reproduces on `main` and `branch_10x`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.