JuliaParallel / JuliaParallel/MPI.jl
CUDA synchronization
Open
CUDA :flower_playing_cards:
needs docs
Open MPI
- Dominant language
- Julia
- Stars
- 439
- Forks
- 127
- Avg merge
- 19h 6m
- Merged PRs (30d)
- 17
Description
The failure on #439 is due to CUDA synchronization issues, From what I understand, the changes in https://github.com/JuliaGPU/CUDA.jl/pull/395 mean that streams are no longer globally synchronized. Since Open MPI operates its own streams, this means that operations can potentially overlap (https://github.com/open-mpi/ompi/issues/7733).
It seems like we need to call `CUDA.synchronize()` (or perhaps `CUDA.synchronize(CuStreamPerThread())`?) before calling MPI?
@vchuravy @maleadt do you have any suggestions/thoughts?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.