[FEA] Allow IVFPQ index object to be constructed with coarse centroids and codebooks
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
We have been getting many requests from users to be able to use IVFPQ's encoding of vectors, but to enable CPU search. Of course, the first recommendation we make for this is to use Faiss for CPU/GPU interoperability but these users want to rely only on cuVS and their own CPU implementations.
We have been discussing a first-class PQ encoder API for some time, but it becomes challenging to be flexible. Many of these users also train the centroids and codebooks once and want to use them across many different shards of IVFPQ indexes. While we still plan to host such an API eventually, an intermediate solution would look something like this:
- User trains coarse centroids and codebooks using cuVS
- User extracts coarse centroids and codebooks from IVFPQ index
- User construct new IVFPQ instance from existing coarse centroids and codebooks
- User calls extend() on IVFPQ instance, extracts the vectors from the lists and copies them into their CPU indexes.
- User uses their CPU version for search
This ultimately comes down to our exposing a constructor for the IVFPQ index that accepts the coarse centroids and codebooks. But we don't want to expose these constructors directly, of course, so we'll need to also expose a new factory function overload for build() in the public APIs that accepts the centroids and codebooks and returns the built IVFPQ index.
This new overload also needs to be exposed through the Python APIs.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing IVFPQ build() public API and its Python bindings, then identify how coarse centroids and codebooks are represented and extracted. Done means a build() overload accepts those inputs, returns a usable IVFPQ index, and the overload is available through the Python API.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, machine-learning, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100