[FEA] Out-of-core kmeans training for coarse quantization and PQ codebooks
Open
@tarang-jain is already working on this.
Since Jan 30, 2026.
feature request
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
Datasets are continuing to grow, and this is making even subsampled workloads for training coarse quantizers and PQ codebooks larger than what some GPUs can fit into their device memory.
We need to come up with a good way to train these centroids and codebooks out of core so that we don't need to force users to have enough device memory available to fit the training dataset.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.