apache / apache/lucene

Exploring GPU based kNN vector search

Open
#13,003 3 comments 3 reactions 0 assignees View on GitHub
type:task vector-based-search
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Through this issue, I wish to explore integrating NVIDIA's kNN indexing and search support, https://github.com/rapidsai/raft. Through our initial benchmarks/prototypes, we found it a lot faster than our HNSW based search.
Along the way, we shall add more details of our experiments through this issue. And will open a PR as soon as something takes shape (right now, things are in extremely early proof-of-concept state).

One of the potential results of this work, if explorations prove worthwhile, can be a Lucene module based on a JNI wrapper around the Raft library.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the experiments and prototypes described in this issue, then compare the proposed NVIDIA RAFT kNN approach with Lucene’s HNSW search. Determine whether a Lucene module using a JNI wrapper is viable; done is a concrete, validated direction or PR scope backed by benchmarks.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.