std::bad_alloc: CUDA error: cudaErrorNotSupported operation not supported in cuvs

Describe the bug
I encountered a std::bad_alloc: CUDA error: cudaErrorNotSupported in cuvs while running segmentation with the RAPIDS cuvs library. The error occurs during the cagra.build step when using a GRID V100DX-32C GPU with CUDA Version 12.0 in a cloud environment. I have tested the same code on several non-virtualized GPUs, and it runs without any issues.

This issue seems to be related to the use of NVIDIA GRID technology for GPU virtualization, but this is only my assumption and might not be correct. The error occurs only on the virtualized GPU (GRID V100DX-32C), while the same code works perfectly on non-virtualized GPUs. The environment is containerized, so the GPU is the only variable between setups.

Could anyone else test this or provide tips on how to investigate further and get to the bottom of this?

Steps/Code to reproduce bug
To reproduce the issue, the following code can be used:

import cupy as cp
import cuvs.neighbors.cagra as cagra

coords_1 = cp.random.rand(100, 3).astype(float32)
coords_2 = cp.random.rand(100, 3).astype(float32)

index_params = cagra.IndexParams(metric="sqeuclidean")
search_params = cagra.SearchParams()

index = cagra.build(index_params, coords_1)
D, I = cagra.search(search_params, index, coords_2, k=10)

During the build step, the error is thrown. Below is the relevant portion of the stack trace for reference:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "resources.pyx", line 110, in cuvs.common.resources.auto_sync_resources.wrapper
  File "cagra.pyx", line 199, in cuvs.neighbors.cagra.cagra.build_index
  File "cagra.pyx", line 200, in cuvs.neighbors.cagra.cagra.build_index
  File "exceptions.pyx", line 37, in cuvs.common.exceptions.check_cuvs
cuvs.common.exceptions.CuvsException: std::bad_alloc: CUDA error at: /__w/cuvs/cuvs/python/cuvs/build/cp310-cp310-manylinux_2_17_x86_64/_deps/rmm-src/include/rmm/mr/device/managed_memory_resource.hpp:64: cudaErrorNotSupported operation not supported

Start with the provided Python reproduction at the cuvs.neighbors.cagra.build call and inspect the stack trace through cuvs/common/resources and the managed_memory_resource.hpp location. Compare behavior on the GRID V100DX-32C and non-virtualized GPUs under CUDA 12.0, then determine whether the failure is specific to managed memory support and document a confirmed cause or workaround.