[BUG] CAGRA build with IVF-PQ build algorithm
@mfoerste4 is already working on this.
Since Feb 3, 2026.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
When building a CAGRA index for the BIGANN-100M dataset using the internal IVF-PQ KNN build algorithm, the process encounters a significant quality issue. The system reports a critically low "Self-included ratio" of 0.39%, including that the initial KNN graph construction is failing to find even most basic neighbor relationships.
Error Message:
Building CAGRA index using IVF-PQ for KNN graph... [982986][22:55:47:687848][warning] Self-included ratio is low: 0.39 %. This can lead to poor recall. Consider using a different configuration for the IVF-PQ index, increasing the refinement rate, or using higher-precision data type for LUT/Internal Distance.
`
Enviornment & Reproduction:
Dataset: BigANN-100M (uint8_t base and query vectors).
Tested Architecture: Hopper (H100)
Current Parameters:
ivf_pq_params.build_params.n_lists = 16384;
ivf_pq_params.build_params.kmeans_n_iters = 25;
ivf_pq_params.build_params.kmeans_trainset_fraction = 0.1;
ivf_pq_params.build_params.conservative_memory_allocation = false;
ivf_pq_params.build_params.pq_bits = 8;
ivf_pq_params.build_params.pq_dim = 64;
ivf_pq_params.search_params.n_probes = 128; )
ivf_pq_params.search_params.lut_dtype = CUDA_R_32F;
ivf_pq_params.search_params.internal_distance_dtype = CUDA_R_32F;
ivf_pq_params.refinement_rate = 8.0;
// Assign to index params
index_params.intermediate_graph_degree = 64;
index_params.graph_degree = 48;
index_params.graph_build_params = ivf_pq_params;
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.