AcademySoftwareFoundation / AcademySoftwareFoundation/openvdb
Issue with DeviceResource (`cudaMallocAsync`) on vGPU without memory pool support
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 3.4k
- Forks
- 777
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 34
Description
When attempting to use the AWS EC2 g6f instance type for CI in fvdb-core (working prototype on this branch), we receive an error at the first call to cudaMallocAsync at nanovdb/cuda/DeviceResource.h:22: CUDA error 801: operation not supported
Through some investigation, it appears that the stream-ordered memory pools capability is not exposed by the vGPU (g6f is a fractional L4 GPU instance). Confirmed when running:
> #include <cuda_runtime.h>
> #include <stdio.h>
>
> int main() {
> int supported = 0, driver = 0;
> cudaDriverGetVersion(&driver);
> cudaError_t e = cudaDeviceGetAttribute(
> &supported, cudaDevAttrMemoryPoolsSupported, 0);
> printf("driver=%d, query=%s, memoryPoolsSupported=%d\n",
> driver, cudaGetErrorString(e), supported);
> return e != cudaSuccess || !supported;
> }
driver=13000, query=no error, memoryPoolsSupported=0
These instances are running the latest AWS-supplied GRID driver (version 19.5/580.159.03) and can confirm the GPU device is available:
>nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03 Driver Version: 580.159.03 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA L4-3Q On | 00000000:31:00.0 Off | 0 |
| N/A N/A P0 N/A / N/A | 0MiB / 3072MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
From reading, it appears that memory pools can be enabled/disabled via hypervisor settings on the vGPU's setup. This seems likely to be disabled due to security concerns; I have opened up an AWS support ticket to confirm if this behaviour is intentional.
Regardless, support for this operating scenario (driver/hardware/CUDA support for cudaMallocAsync but unavailable capability) would require utilizing cudaMalloc (or a custom allocator) instead. This is a related issue in the work for #2232 and allowing NanoVDB to use an alternative allocator to enable successful execution on a vGPU configured in this way could be acceptance criteria for that work.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at nanovdb/cuda/DeviceResource.h:22 and inspect how the first cudaMallocAsync call is used. Compare the proposed allocator work in #2232 with the g6f/vGPU capability result; done means the CI prototype can allocate successfully when memoryPoolsSupported=0 without relying on unsupported cudaMallocAsync.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, hpc
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100