Customize Distance Calculations for KNN Vectors
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Today I see 2 ways to provide the distance calculations when using the HNSW vectors in Lucene:
1. The existing `VectorSimilarityFunction`, which is encoded into the segment file itself.
2. Via a customer scorer through a custom `KnnVectorsFormat`.
IMO this is not a great experience because in order to provide my own scorer I need to implement at least 2 new classes but for the most part the code in those classes would be boilerplate/duplicated code. In fact really the only novel code there would be in `RandomVectorScorer#score`. I do see that we're a little bit stuck with this because the existing `VectorSimilarityFunction` class is implemented as an enum so we can't extend it (or really make any changes to it).
I see that adding bit/binary vector support (https://github.com/apache/lucene/issues/13505) is also currently blocked on resolving this, so I wanted to ask:
1. What's the remaining gap to officially supporting bit vectors in Lucene? Naively it looks as simple as moving the new `HnswBitVectorsFormat` class introduced in #13288 into the `lucene101` package.
2. Broadly speaking what is the vision here for allowing users to customize the distance calculations? For example does the current approach with implementing a custom format/scorer look like the longer term strategy or instead the long term plan look something like replacing `VectorSimilarityFunction` with an extensible interface instead?
Contributor guide
Research direction
Start by reading VectorSimilarityFunction, KnnVectorsFormat, RandomVectorScorer#score, and the HnswBitVectorsFormat discussion linked from issues #13505 and #13288. This issue asks for design direction on bit-vector support and customizable distance calculations, so it does not define a specific implementation or completion test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100