[CUDA][HIP] too many process spawned on multiple GPU systems
- Dominant language
- LLVM
- Stars
- 1.5k
- Forks
- 854
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 137
Description
### Describe the bug
On multiple GPU systems, using HIP or CUDA, a process is spawned on all GPUs instead being spawned only on one of them. (See To reproduce section)
This result in memory leaks when SYCL is used with both mpich and openmpi as both GPUs ends up receiving the data, even though the program (in the following exemple a private HPC application) only use one of them per MPI ranks. This result in a graph like this (memory usage per process / time)
`mpirun -n 2 <...>`

where the blue and red curve are the working GPU processes, and the two other growing ones are the threads on the wrong GPUs.
CUDA_VISIBLE_DEVICES can be used to circumvent the issue
```
mpirun \
-n 1 -x CUDA_VISIBLE_DEVICES=0 <...> : \
-n 1 -x CUDA_VISIBLE_DEVICES=1 <...>
```

### To reproduce
```c++
#include
#include
std::vector get_sycl_device_list() {
std::vector devs;
const auto &Platforms = sycl::platform::get_platforms();
for (const auto &Platform : Platforms) {
const auto &Devices = Platform.get_devices();
for (const auto &Device : Devices) {
devs.push_back(Device);
return devs;
}
}
return devs;
}
int main(void){
for (auto d : get_sycl_device_list()){
auto DeviceName = d.get_info();
std::cout <
Contributor guide
Assessment
This issue has not been assessed yet.