NVIDIA / NVIDIA/cuvs

[BUG] Calculating ground truth on GPU in cuvs-bench fails in 26.04 docker image

Open
#2,050 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

Failed to compute ground truth in cuvs-bench 26.04. Same command worked previously in 26.02.

docker run --gpus all --rm           --user root           --shm-size=16GB           --entrypoint /bin/bash           --workdir /data/benchmarks           -v /raid/cuvs-bench-runner:/data/benchmarks           rapidsai/cuvs-bench:26.04-cuda13-py3.12 -c "python -m cuvs_bench.generate_groundtruth         /data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M/base.fbin         --queries=/data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M/query.fbin         --output /data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M -k 10
        "
          
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/conda/lib/python3.12/site-packages/cuvs_bench/generate_groundtruth/__main__.py", line 58, in <module>
    xp = import_with_fallback("cupy", "numpy")
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/cuvs_bench/generate_groundtruth/__main__.py", line 44, in import_with_fallback
    module = importlib.import_module(primary_lib)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 999, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/opt/conda/lib/python3.12/site-packages/cupy/__init__.py", line 18, in <module>
    from cupy import _core  # NOQA
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/cupy/_core/__init__.py", line 5, in <module>
    from cupy._core import core  # NOQA
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "cupy/_core/core.pyx", line 1, in init cupy._core.core
  File "cupy/_core/_routines_math.pyx", line 7, in init cupy._core._routines_math
  File "cupy/_core/_reduction.pyx", line 1, in init cupy._core._reduction
  File "cupy/_core/_cub_reduction.pyx", line 260, in init cupy._core._cub_reduction
  File "/opt/conda/lib/python3.12/site-packages/cupy/_environment.py", line 86, in get_nvcc_path
    _nvcc_path = _get_nvcc_path()
                 ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/cupy/_environment.py", line 198, in _get_nvcc_path
    cuda_path = get_cuda_path()
                ^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/cupy/_environment.py", line 78, in get_cuda_path
    _cuda_path = _get_cuda_path()
                 ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/cupy/_environment.py", line 151, in _get_cuda_path
    conda_cuda_path = _get_conda_cuda_path()
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/cupy/_environment.py", line 130, in _get_conda_cuda_path
    cuda_path = os.path.join(conda_prefix, 'targets', arch)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 76, in join
TypeError: expected str, bytes or os.PathLike object, not NoneType

Works in 26.02:

docker run --gpus all --rm           --user root           --shm-size=16GB           --entrypoint /bin/bash           --workdir /data/benchmarks           -v /raid/cuvs-bench-runner:/data/benchmarks           rapidsai/cuvs-bench:26.02-cuda13-py3.12 -c "python -m cuvs_bench.generate_groundtruth         /data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M/base.fbin         --queries=/data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M/query.fbin         --output /data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M -k 10
        "
          
Reading whole dataset
Dataset size    4.6 GB, shape (1500000, 768), dtype float32
Reading queries from file /data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M/query.fbin
Calculating true nearest neighbors
Step 0/3:
Step 1/3:
Step 2/3:
writing /data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M/groundtruth.neighbors.ibin (10000, 10) uint32 ...
writing /data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M/groundtruth.distances.fbin (10000, 10) float32 ...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with cuvs_bench.generate_groundtruth/main.py, especially import_with_fallback, and compare the 26.04 and 26.02 CUDA 13 Python 3.12 images. Reproduce the provided docker command and trace the CuPy CUDA-path failure. Done means the 26.04 GPU command computes and writes the ground-truth files as shown for 26.02.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
cli, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.