intel / intel/llvm

[CUDA][HIP] too many process spawned on multiple GPU systems

Open
#15,251 9 comments 0 reactions 0 assignees View on GitHub
bug cuda hip
Dominant language
LLVM
Stars
1.5k
Forks
854
Avg merge
3d 17h
Merged PRs (30d)
137

Description

### Describe the bug

On multiple GPU systems, using HIP or CUDA, a process is spawned on all GPUs instead being spawned only on one of them. (See To reproduce section)

This result in memory leaks when SYCL is used with both mpich and openmpi as both GPUs ends up receiving the data, even though the program (in the following exemple a private HPC application) only use one of them per MPI ranks. This result in a graph like this (memory usage per process / time)
`mpirun -n 2 <...>`
![Screenshot_2024-09-01_21-14-17](https://github.com/user-attachments/assets/1b18e147-9d12-4a80-972c-166c95d64cbd)
where the blue and red curve are the working GPU processes, and the two other growing ones are the threads on the wrong GPUs.

CUDA_VISIBLE_DEVICES can be used to circumvent the issue
```
mpirun \
-n 1 -x CUDA_VISIBLE_DEVICES=0 <...> : \
-n 1 -x CUDA_VISIBLE_DEVICES=1 <...>
```
![Screenshot_2024-09-01_21-45-25](https://github.com/user-attachments/assets/dea05663-6447-493c-92a3-889be73f7dd9)

### To reproduce

```c++
#include
#include

std::vector get_sycl_device_list() {
std::vector devs;
const auto &Platforms = sycl::platform::get_platforms();
for (const auto &Platform : Platforms) {
const auto &Devices = Platform.get_devices();
for (const auto &Device : Devices) {
devs.push_back(Device);
return devs;
}
}
return devs;
}

int main(void){

for (auto d : get_sycl_device_list()){
auto DeviceName = d.get_info();
std::cout <

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.