apache / apache/lucene

[Feature] Add support for passing extra information with KNNVectorField

Open
#14,247 8 comments 0 reactions 0 assignees View on GitHub
vector-based-search
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

## Description
Currently KnnFloatVectorField(float/byte) supports only passing Vector information(which includes dim etc). But sometime you need to pass some more extra information with vector the vector field which will be helpful in building the efficient search data structures. A naive example for this can be a tenant id where you might want to cluster the points specific to a tenant in the hnsw graph.

The ask for this issue is to have a byte array/or some thing similar that can be passed with the vector field and can be made accessible at the KNNVectorsFormat level.

## Alternatives
I tried passing the information in another field like a BinaryDocValues, but during indexing at the KNNVectorsFormat we don't have access to the BDV. If there is an alternative way to access which I might be missing, please let me know.

Contributor guide

Open the contributing guide

Research direction

Start by tracing KnnFloatVectorField and KNNVectorsFormat to understand how vector information currently reaches the format layer. Investigate the proposed byte-array or equivalent metadata path and whether an existing field can provide it during indexing. Done should mean a decided API and format-level access pattern, covered by relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.