multinode benchmarks fail to start with error - [CUDA_ERROR_INVALID_VALUE] invalid argument in expression cuMemSetAccess
Open
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 778
- Forks
- 93
- PR merge metrics
- No merged PRs in 30d
Description
Running multinode benchmarks works on single node, but fails with the following error when running on multiple nodes:
mpirun --map-by ppr:8:node --bind-to core -np 16 --report-bindings --hostfile hostfile /usr/local/bin/nvbandwidth -p multinode
...
Running multinode_device_to_device_memcpy_read_ce.
[CUDA_ERROR_INVALID_VALUE] invalid argument in expression cuMemSetAccess((CUdeviceptr) buffer, roundedUpAllocationSize, &desc, 1 ) on gpub200-nom6ae0518, rank = 8 in MultinodeMemoryAllocationUnicast::MultinodeMemoryAllocationUnicast(size_t, int)() : /home/code/external/nvbandwidth/multinode_memcpy.cpp:76
Binary was build with Cuda 12.8
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the multinode benchmark with the shown mpirun command and CUDA 12.8, then inspect external/nvbandwidth/multinode_memcpy.cpp at line 76 where cuMemSetAccess fails. Compare the single-node and multinode paths and verify that the multinode benchmark starts and completes without CUDA_ERROR_INVALID_VALUE.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100