NVIDIA / NVIDIA/cuvs

[BUG] Error while building the Cagra index with IVF PQ with error too many invalid or duplicated neighbor nodes

Open
#2,209 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

Describe the bug
While building the cagra index using Faiss we are seeing this error

C++ exception RAFT failure at graph_core.cuh line=1417:
Could not generate an intermediate CAGRA graph because the initial kNN graph
contains too many invalid or duplicated neighbor nodes.

Some environment details:

libcuvs=25.08 
cuda-nvcc 'cuda-version>=12.0,<=12.5' 
cuda-toolkit=12.4.1 
gxx_linux-64=12.4

Faiss commit: https://github.com/facebookresearch/faiss/tree/f9ccd582f9a9b8400428625d1ff3217ae83422b9

Job Failure details

Parameter Value
Dimension 1024
Total Docs (vectors) 27,326
Data Type float32
Space Type L2 (Euclidean)

CAGRA Index Configuration

Parameter Value Derivation
graph_degree 32 m * 2 → 16 × 2
intermediate_graph_degree 64 m * 4 → 16 × 4
graph_build_algo IVF_PQ Default for float data
n_lists 165 int(sqrt(doc_count)) → int(sqrt(27326))
pq_dim 256 dimension / 4 → 1024 / 4
refine_rate 1.0 Default (no refinement pass)
store_dataset False Default
device 0 Default (single GPU)

Error

RAFT failure at file=cpp/src/neighbors/detail/cagra/graph_core.cuh line=1417:
Could not generate an intermediate CAGRA graph because the initial kNN graph
contains too many invalid or duplicated neighbor nodes. This error can occur,
for example, if too many overflows occur during the norm computation between
the dataset vectors.

Warning Observed Before Failure

[warning] Self-included ratio is low: 4.41%. This can lead to poor recall.
Consider using a different configuration for the IVF-PQ index, increasing the
refinement rate, or using higher-precision data type for LUT/Internal Distance.

Call Stack

faiss.GpuIndexCagra::trainEx()
  → cuvs::neighbors::cagra::build()
    → cuvs::neighbors::cagra::detail::build()
      → cuvs::neighbors::cagra::detail::graph::optimize()  ← FAILS HERE (line 1417)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at cpp/src/neighbors/detail/cagra/graph_core.cuh line 1417 and trace the graph optimize path shown in the call stack. Reproduce the failure with the reported 1,024-dimensional, 27,326-vector IVF_PQ configuration, then determine why invalid or duplicated neighbors are produced alongside the low self-included ratio. Done means the configuration no longer fails and the relevant behavior has regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.