AcademySoftwareFoundation / AcademySoftwareFoundation/openvdb

Issue with DeviceResource (`cudaMallocAsync`) on vGPU without memory pool support

Open
#2,255 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug nanovdb
Dominant language
C++
Stars
3.4k
Forks
777
Avg merge
3d 9h
Merged PRs (30d)
34

Description

When attempting to use the AWS EC2 g6f instance type for CI in fvdb-core (working prototype on this branch), we receive an error at the first call to cudaMallocAsync at nanovdb/cuda/DeviceResource.h:22: CUDA error 801: operation not supported

Through some investigation, it appears that the stream-ordered memory pools capability is not exposed by the vGPU (g6f is a fractional L4 GPU instance). Confirmed when running:

> #include <cuda_runtime.h>
> #include <stdio.h>
>
> int main() {
>   int supported = 0, driver = 0;
>   cudaDriverGetVersion(&driver);
>   cudaError_t e = cudaDeviceGetAttribute(
>       &supported, cudaDevAttrMemoryPoolsSupported, 0);
>   printf("driver=%d, query=%s, memoryPoolsSupported=%d\n",
>          driver, cudaGetErrorString(e), supported);
>   return e != cudaSuccess || !supported;
> }

driver=13000, query=no error, memoryPoolsSupported=0

These instances are running the latest AWS-supplied GRID driver (version 19.5/580.159.03) and can confirm the GPU device is available:

>nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03             Driver Version: 580.159.03     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4-3Q                   On  |   00000000:31:00.0 Off |                    0 |
| N/A   N/A    P0            N/A  /  N/A  |       0MiB /   3072MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

From reading, it appears that memory pools can be enabled/disabled via hypervisor settings on the vGPU's setup. This seems likely to be disabled due to security concerns; I have opened up an AWS support ticket to confirm if this behaviour is intentional.

Regardless, support for this operating scenario (driver/hardware/CUDA support for cudaMallocAsync but unavailable capability) would require utilizing cudaMalloc (or a custom allocator) instead. This is a related issue in the work for #2232 and allowing NanoVDB to use an alternative allocator to enable successful execution on a vGPU configured in this way could be acceptance criteria for that work.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at nanovdb/cuda/DeviceResource.h:22 and inspect how the first cudaMallocAsync call is used. Compare the proposed allocator work in #2232 with the g6f/vGPU capability result; done means the CI prototype can allocate successfully when memoryPoolsSupported=0 without relying on unsupported cudaMallocAsync.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, hpc
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.