[FEA] Option to train Vamana OPQ codebooks and rotation matrix
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
Is your feature request related to a problem? Please describe.
Current (initial) version of Vamana disk index build (with quantization) requires pre-trained codebooks. This is for initial use cases, but future users may want to create codebooks within cuVS.
Describe the solution you'd like
A separate API call to train OPQ codebooks and rotation matrix that can easily be used with vamana::build. Also need the ability to serialize these so that they can be used with search through the CPU DiskANN library.
Describe alternatives you've considered
Another option is to have a flag that creates the codebooks during vamana::build, but this gives much less flexibility and complicates the build call.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the vamana::build entry point and reviewing how pre-trained OPQ codebooks are currently supplied. Define the separate training API and serialization path needed for codebooks and the rotation matrix, then verify that the serialized output can be consumed by the CPU DiskANN library.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100