NVIDIA / NVIDIA/cuvs

[BUG] cuVS Bench is not working with Deep 100M dataset

Open
#1,407 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

Describe the bug
I'm following the instruction in https://docs.rapids.ai/api/cuvs/nightly/cuvs_bench/#end-to-end-large-scale-benchmarks-10m-vectors to run cuVS Bench with Deep-100M dataset.

Steps/Code to reproduce bug

  1. conda create --name cuvs_benchmarks_deep100m
  2. conda activate cuvs_benchmarks_deep100m
  3. conda install -c rapidsai -c conda-forge cuvs-bench=25.08 cuda-version=13.0*
  4. conda install -c rapidsai -c conda-forge cuvs-bench=25.08 cuda-version=12.0*
  5. python3 -m cuvs_bench.run --dataset deep-1B --algorithms cuvs_cagra --batch-size 10 -k 10

The deep dataset was downloaded manually and the groundtruth file was split into two files per the instructions in the cuVS Bench documentation.

Expected behavior
The graphs should get generated. After that, the Search should be successful. But, I find that the graph construction is stuck for a long time. I waited for > 24 hrs and step# 5 seems to be hung. From quick debugging, I found the kernel kern_sort is continuously running.

Environment details (please complete the following information):

  • Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]
  • Method of RAFT install: [conda, Docker, or from source]
    • If method of install is [Docker], provide docker pull & docker run commands used
      GPU : A100 80GB.
      CUDA ver: 13.0
      Driver Version: 580.82.07
      Installation method: Conda
      Environment: Bare Metal

Additional context
Add any other context about the problem here.
The Deep-100M dataset was working with the former RAFT Bench.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the documented Deep-100M benchmark with the python3 -m cuvs_bench.run entry point and the reported cuvs_cagra algorithm. Inspect the graph-construction path around the continuously running kern_sort kernel, comparing it with the former RAFT Bench behavior. Done means graph construction completes and the subsequent search succeeds for the reported environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.