NVIDIA / NVIDIA/cuvs

[BUG] CAGRA may miss exact self-match in IP self-query

Open
#2,102 13 comments 0 reactions 1 assignee View on GitHub

@irina-resh-nvda is already working on this.

Since Jul 21, 2026.

bug
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

Describe the bug

Hi cuVS team, thanks for taking a look.

I'm seeing cuvs.neighbors.cagra miss exact self-matches in a small inner-product self-query repro. I'm not sure whether this is expected for this CAGRA search configuration, but it is surprising for downstream usage because the exact same vector is present in the indexed dataset.

With normalized vectors, querying xb[i] against an index built from xb has an exact self score close to 1.0. In the repro below, some queries return a different vector with score around 0.16-0.21, and the self vector is not present in the returned top-k.

Steps/Code to reproduce bug

import numpy as np
import cupy as cp
from cuvs.neighbors import cagra

N = 50_000
D = 384
Q = 512
K = 10

np.random.seed(0)
xb_h = np.random.randn(N, D).astype(np.float32)
xb_h /= np.linalg.norm(xb_h, axis=1, keepdims=True)
xb = cp.asarray(xb_h)

index = cagra.build(
    cagra.IndexParams(
        metric="inner_product",
        intermediate_graph_degree=64,
        graph_degree=32,
        build_algo="ivf_pq",
    ),
    xb,
)

dist, ind = cagra.search(
    cagra.SearchParams(
        itopk_size=64,
        search_width=1,
    ),
    index,
    xb[:Q],
    K,
)

dist_h = cp.asnumpy(dist)
ind_h = cp.asnumpy(ind)

expected = np.arange(Q)
bad = np.flatnonzero(ind_h[:, 0] != expected)
print(f"self_top1_count={Q - bad.size}/{Q}")
print(f"first_bad_queries={bad[:10].tolist()}")

for q in bad[:5]:
    top_id = int(ind_h[q, 0])
    returned_score = float(dist_h[q, 0])
    manual_returned_score = float(np.dot(xb_h[q], xb_h[top_id]))
    manual_self_score = float(np.dot(xb_h[q], xb_h[q]))
    print(
        {
            "query": int(q),
            "top_id": top_id,
            "returned_score": returned_score,
            "manual_returned_score": manual_returned_score,
            "manual_self_score": manual_self_score,
        }
    )

Observed output on my machine:

self_top1_count=488/512
first_bad_queries=[3, 27, 38, 68, 135, 138, 144, 182, 206, 238]
{'query': 3, 'top_id': 15297, 'returned_score': 0.18928107619285583, 'manual_returned_score': 0.18928107619285583, 'manual_self_score': 1.0}
{'query': 27, 'top_id': 34697, 'returned_score': 0.210464745759964, 'manual_returned_score': 0.210464745759964, 'manual_self_score': 0.9999999403953552}
{'query': 38, 'top_id': 6944, 'returned_score': 0.16181255877017975, 'manual_returned_score': 0.16181255877017975, 'manual_self_score': 0.9999999403953552}

I also observed the same kind of miss with a smaller row count, for example N=12_000 produced self_top1_count=511/512 with the same script shape.

Expected behavior

Expected the exact self vector to be returned for these self-queries, with an inner-product score close to 1.0.

If this is expected behavior for these CAGRA parameters, any guidance on the recommended build/search settings for this pattern would be appreciated.

Environment details (please complete the following information):

  • Environment location: bare-metal
  • Method of cuVS install: PyPI wheel
  • cuVS version: 25.06.01
  • CuPy version: 14.0.1
  • GPU: NVIDIA L4
  • Driver: 560.35.05
  • CUDA runtime reported by nvidia-smi: 12.6

Additional context

This came up while investigating a downstream Milvus GPU_CAGRA report: https://github.com/milvus-io/milvus/issues/49864

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.