NVIDIA / NVIDIA/cuvs

[BUG] `cuvs-bench` should not fail when the GPU package is installed on a CPU node

Open
#1,206 0 comments 0 reactions 1 assignee View on GitHub

@jnke2016 is already working on this.

Since Nov 20, 2025.

bug
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

We recently had a user install the GPU cuvs-bench package on a CPU node that didn't have a GPU. Docker was happy to install the cuda toolkit and runtime, even though no GPU was present. There are some stubs in cuvs-bench for the cuda runtime calls that need to be made, and it's smartly checking to see if cuda runtime is available before making said calls so that the CPU package can run the same code paths on hardware without GPUs.

The problem is that when the cuda runtime IS actually installed but a GPU is not, the actual cuda runtime throws errors that are unchecked and cause nasty memory issues downstream that ultimately end up manifesting as segfaults. We now issue a warning in the logs so the user has an idea of why the package might be failing, but we should also be checking these cuda runtime calls (specifically the calls to cudaGetDeviceProps) and not failing altogether.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.