[BUG] Calculating ground truth on GPU in cuvs-bench fails in 26.04 docker image
Open
Nobody has claimed this yet.
bug
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
Failed to compute ground truth in cuvs-bench 26.04. Same command worked previously in 26.02.
docker run --gpus all --rm --user root --shm-size=16GB --entrypoint /bin/bash --workdir /data/benchmarks -v /raid/cuvs-bench-runner:/data/benchmarks rapidsai/cuvs-bench:26.04-cuda13-py3.12 -c "python -m cuvs_bench.generate_groundtruth /data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M/base.fbin --queries=/data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M/query.fbin --output /data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M -k 10
"
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/opt/conda/lib/python3.12/site-packages/cuvs_bench/generate_groundtruth/__main__.py", line 58, in <module>
xp = import_with_fallback("cupy", "numpy")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.12/site-packages/cuvs_bench/generate_groundtruth/__main__.py", line 44, in import_with_fallback
module = importlib.import_module(primary_lib)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 999, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/opt/conda/lib/python3.12/site-packages/cupy/__init__.py", line 18, in <module>
from cupy import _core # NOQA
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.12/site-packages/cupy/_core/__init__.py", line 5, in <module>
from cupy._core import core # NOQA
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "cupy/_core/core.pyx", line 1, in init cupy._core.core
File "cupy/_core/_routines_math.pyx", line 7, in init cupy._core._routines_math
File "cupy/_core/_reduction.pyx", line 1, in init cupy._core._reduction
File "cupy/_core/_cub_reduction.pyx", line 260, in init cupy._core._cub_reduction
File "/opt/conda/lib/python3.12/site-packages/cupy/_environment.py", line 86, in get_nvcc_path
_nvcc_path = _get_nvcc_path()
^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.12/site-packages/cupy/_environment.py", line 198, in _get_nvcc_path
cuda_path = get_cuda_path()
^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.12/site-packages/cupy/_environment.py", line 78, in get_cuda_path
_cuda_path = _get_cuda_path()
^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.12/site-packages/cupy/_environment.py", line 151, in _get_cuda_path
conda_cuda_path = _get_conda_cuda_path()
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.12/site-packages/cupy/_environment.py", line 130, in _get_conda_cuda_path
cuda_path = os.path.join(conda_prefix, 'targets', arch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen posixpath>", line 76, in join
TypeError: expected str, bytes or os.PathLike object, not NoneType
Works in 26.02:
docker run --gpus all --rm --user root --shm-size=16GB --entrypoint /bin/bash --workdir /data/benchmarks -v /raid/cuvs-bench-runner:/data/benchmarks rapidsai/cuvs-bench:26.02-cuda13-py3.12 -c "python -m cuvs_bench.generate_groundtruth /data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M/base.fbin --queries=/data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M/query.fbin --output /data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M -k 10
"
Reading whole dataset
Dataset size 4.6 GB, shape (1500000, 768), dtype float32
Reading queries from file /data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M/query.fbin
Calculating true nearest neighbors
Step 0/3:
Step 1/3:
Step 2/3:
writing /data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M/groundtruth.neighbors.ibin (10000, 10) uint32 ...
writing /data/benchmarks/datasets/BeIR-trec-news-generated-queries-1.5M/groundtruth.distances.fbin (10000, 10) float32 ...
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with cuvs_bench.generate_groundtruth/main.py, especially import_with_fallback, and compare the 26.04 and 26.02 CUDA 13 Python 3.12 images. Reproduce the provided docker command and trace the CuPy CUDA-path failure. Done means the 26.04 GPU command computes and writes the ground-truth files as shown for 26.02.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- cli, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100