Padded dataset can affect IVF-PQ encoding
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
We are allowing users to pass in padded_dataset types to CAGRA build.
In the IVF-PQ CAGRA build option, stride is used to interpret the number of dimensions of the dataset.
The stride includes extra zeros set by the padding.
This can cause the PQ encoding and heuristics to be different. For example the derived pq_len would be different since pq_len = dim (stride) / pq_dim. That would cause the codebooks to look different.
Moreover, it can change whether we apply rotation matrix or not. dim (stride) % pq_dim == 0.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the IVF-PQ CAGRA build path that handles padded_dataset and uses stride as the dataset dimension. Compare the logical dimension with the padded stride and verify the resulting pq_len, codebooks, heuristics, and rotation decision; done means padding no longer changes these outcomes.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100