NVIDIA / NVIDIA/cuda-python

[BUG]: VirtualMemoryResource `host_numa` allocations always fail

Open
#2,694 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triage
Dominant language
Cython
Stars
3.4k
Forks
329
Avg merge
1d 23h
Merged PRs (30d)
116

Description

Is this a duplicate?
Type of Bug

Runtime Error

Component

cuda.core

Describe the bug

Every VirtualMemoryResource allocation with location_type="host_numa" fails with CUDA_ERROR_INVALID_VALUE, on a system that reports host_numa_id 0 and host_numa_virtual_memory_management_supported True. VirtualMemoryResource.allocate passes prop.location.id = -1 whenever the resource has no device, which __init__ sets for every host location type, and cuMemCreate rejects CU_MEM_LOCATION_TYPE_HOST_NUMA with that id. cuda_core/cuda/core/typing.py documents HOST_NUMA as "host memory pinned to a specific NUMA node", and VirtualMemoryResourceOptions has no field that carries a node id.

location_type="host_numa_current" also fails, for a separate reason: cuMemCreate rejects CU_MEM_LOCATION_TYPE_HOST_NUMA_CURRENT for every node id I passed it, including 0.

test_vmm_host_location_types_report_host_accessible in cuda_core/tests/test_memory.py parametrizes over host, host_numa and host_numa_current, but it only checks mr.device and mr.is_host_accessible, so it stays green without ever calling allocate.

How to Reproduce
from cuda.core import Device, VirtualMemoryResource, VirtualMemoryResourceOptions

dev = Device()
dev.set_current()
print("host_numa_id =", dev.properties.host_numa_id)
for location_type in ("host", "host_numa"):
    opts = VirtualMemoryResourceOptions(location_type=location_type, handle_type=None)
    buf = VirtualMemoryResource(dev, config=opts).allocate(4096)
    print(location_type, "allocated", buf.size, "bytes")
    buf.close()
host_numa_id = 0
host allocated 2097152 bytes
Traceback (most recent call last):
  File "/tmp/vmm_host_numa.py", line 8, in <module>
    buf = VirtualMemoryResource(dev, config=opts).allocate(4096)
  File "/home/vyron-vasileiadis/projects/forks/cuda-python/cuda_core/cuda/core/_memory/_virtual_memory_resource.py", line 555, in allocate
    raise_if_driver_error(res)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "cuda/core/_utils/cuda_utils.pyx", line 134, in cuda.core._utils.cuda_utils._check_driver_error
  File "cuda/core/_utils/cuda_utils.pyx", line 145, in cuda.core._utils.cuda_utils._check_driver_error
cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.
Expected behavior

allocate should either return a buffer pinned to a NUMA node, the way location_type="host" and location_type="device" already do, or fail at construction with an error naming the missing node id. Driving cuMemCreate directly on this system, CU_MEM_LOCATION_TYPE_HOST_NUMA returns CUDA_SUCCESS at location.id = 0 and CUDA_ERROR_INVALID_VALUE at location.id = -1, so the location id is the only thing standing between the current behaviour and a working allocation.

Operating System

Ubuntu 26.04 LTS

nvidia-smi output
Tue Aug 25 11:22:23 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.84                 Driver Version: 595.84         CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3050 ...    Off |   00000000:01:00.0 Off |                  N/A |
| N/A   46C    P8              3W /   30W |      11MiB /   4096MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in cuda_core/cuda/core/_memory/_virtual_memory_resource.py, especially VirtualMemoryResource.init and allocate, and review the HOST_NUMA definitions in cuda_core/cuda/core/typing.py. Run cuda_core/tests/test_memory.py and extend the host-location coverage to exercise allocation, including the reported host_numa and host_numa_current cases. Done means the supported host location behavior is tested and allocation no longer fails because the location id is missing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.