apache / apache/lucene

Should we have an interface VectorValues which would be implemented by [Byte/Float]VectorValues classes

Open
#12,635 0 comments 0 reactions 0 assignees View on GitHub
type:enhancement vector-based-search
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Currently, there is lot of code duplication due to [ByteVectorValues](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/ByteVectorValues.java) and [FloatVectorValues](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/FloatVectorValues.java) (Example - [one](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsWriter.java#L48-L73), [two](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsWriter.java#L109-L301) etc.) because they are handled separately as one returns byte[] and other float[] for `vectorValue` method. The idea is if we should have a common interface `VectorValues`? that provides the common functionality and we could eventually get rid of this duplication. As a first step maybe we could try to move the below methods(common in above 2 classes) into its different interface `VectorValues`?

- int dimension()
- int size()
- byte[]/float[] vectorValue()

There could be some other ideas as well(maybe better) to address the code duplication here :

1. Making `DISI` implementing an interface which `VectorValues`(interface in [this PR](https://github.com/apache/lucene/pull/12636)) would extend making further room to remove duplication (or)
2. Making a wrapper over `ByteVectorValues` and `FloatVectorValues` etc?.

Looking for suggestions or more ideas on how to better handle this and avoid this current duplication.

Contributor guide

Open the contributing guide

Research direction

Start by comparing ByteVectorValues.java and FloatVectorValues.java, then inspect the duplicated paths in KnnVectorsWriter.java and the VectorValues interface proposed in pull request 12636. Review the listed common methods and the alternative designs in the issue; done would require an agreed approach that reduces the duplication without losing the byte[] and float[] vector behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.