Integrate a JVector codec for KNN searches
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
JVector (https://github.com/datastax/jvector) is "a graph-based index that builds on the HNSW and DiskANN designs with composable extensions."
JVector features a DiskANN implementation and allows for multi-phase vector matching and disk-based rerank. The need for JVector as a search engine is neatly summarized in this OpenSearch [issue](https://github.com/opensearch-project/k-NN/issues/2386).
Proposing a new codec in Lucene as a standalone `KnnVectorsFormat` that is based on [OpenSearch's implementation](https://github.com/opensearch-project/opensearch-jvector) of JVector. This implementation would integrate with Lucene's existing vector APIs and codec SPI.
Opening this issue to get feedback on the need, implementation, and long term considerations for this codec.
Contributor guide
Research direction
Start by reviewing Lucene's existing KnnVectorsFormat APIs and codec SPI, then compare them with OpenSearch's opensearch-jvector implementation. The issue is seeking feedback on the need, implementation, and long-term considerations; done would require an agreed design for a standalone JVector-based codec, but no files or tests are named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100