NVIDIA / NVIDIA/cuvs

[BUG] Illegal Memory Access Issue with CAGRA Search with Pre-filtering

Open
#1,022 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

Describe the bug
I am running cuVS CAGRA filtered search with 5M vectors, 64 dim, fp16. Running a grid search and with a specific parameter set, filtering percentage, and batch size I get the below error-

Error in worker: cudaErrorIllegalAddress: an illegal memory access was encountered
CUDA call='cudaFreeAsync(ptr, stream)' at file=/__w/cuvs/cuvs/cpp/src/neighbors/detail/cagra/compute_distance.hpp line=259 failed with an illegal memory access was encountered
Error in worker: CUDA error encountered at: file=/pyenv/versions/3.13.3/lib/python3.13/site-packages/libraft/include/raft/core/interruptible.hpp line=303: call='query_result', Reason=cudaErrorIllegalAddress:an illegal memory access was encountered
Obtained 24 stack frames

Parameters - graph_degree: 64; intermediate_graph_degree: 64, itopk_size: 64, search_width: 8
Filtering percentage: 16%
Query Batch Size: 1000

When I don't filter, there is no illegal memory access issue.

Steps/Code to reproduce bug
Follow the steps here

Expected behavior

  1. No illegal memory address issue.

Environment details (please complete the following information):

  • AWS (g6e.8xlarge - 1 L40S, 32 vCPU, 256GB RAM)
  • cuVS 25.06
  • pip install

Additional Context
Attaching the compute-sanitizer log for illegal memory access issue with query_batch_size = 1000 for some parameter.

memcheck_illegal_access (1).log

The log is generated for the parameters: graph_degree: 64; intermediate_graph_degree: 64, itopk_size: 64, search_width: 8

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the compute-sanitizer log and cuVS CAGRA's cuvs/cpp/src/neighbors/detail/cagra/compute_distance.hpp at line 259. Reproduce using the linked steps with the listed graph, filtering, search-width, and query-batch parameters, comparing filtered and unfiltered search. Done means the filtered search completes without an illegal memory access.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
performance, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.