apache / apache/lucene

Customize Distance Calculations for KNN Vectors

Open
#14,025 0 comments 0 reactions 0 assignees View on GitHub
type:enhancement vector-based-search
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Today I see 2 ways to provide the distance calculations when using the HNSW vectors in Lucene:
1. The existing `VectorSimilarityFunction`, which is encoded into the segment file itself.
2. Via a customer scorer through a custom `KnnVectorsFormat`.

IMO this is not a great experience because in order to provide my own scorer I need to implement at least 2 new classes but for the most part the code in those classes would be boilerplate/duplicated code. In fact really the only novel code there would be in `RandomVectorScorer#score`. I do see that we're a little bit stuck with this because the existing `VectorSimilarityFunction` class is implemented as an enum so we can't extend it (or really make any changes to it).

I see that adding bit/binary vector support (https://github.com/apache/lucene/issues/13505) is also currently blocked on resolving this, so I wanted to ask:
1. What's the remaining gap to officially supporting bit vectors in Lucene? Naively it looks as simple as moving the new `HnswBitVectorsFormat` class introduced in #13288 into the `lucene101` package.
2. Broadly speaking what is the vision here for allowing users to customize the distance calculations? For example does the current approach with implementing a custom format/scorer look like the longer term strategy or instead the long term plan look something like replacing `VectorSimilarityFunction` with an extensible interface instead?

Contributor guide

Open the contributing guide

Research direction

Start by reading VectorSimilarityFunction, KnnVectorsFormat, RandomVectorScorer#score, and the HnswBitVectorsFormat discussion linked from issues #13505 and #13288. This issue asks for design direction on bit-vector support and customizable distance calculations, so it does not define a specific implementation or completion test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.