JuliaParallel / JuliaParallel/MPI.jl
MPI implementations intercepting Signals is incompatible with Julia GC safepoint
- Dominant language
- Julia
- Stars
- 439
- Forks
- 127
- Avg merge
- 19h 6m
- Merged PRs (30d)
- 17
Description
Thanks again for your help with https://github.com/JuliaParallel/MPI.jl/issues/720 - this one is unrelated (except that issue #720 lead us to create more comprehensive unit test revealing this new, probably unrelated segfault).
**Summary of this problem:** a segfault occurs when GC is triggered in a multithreaded+MPI context.
**How to reproduce:** I have create a draft PR adding a GC.gc() call in one of MPI.jl's existing multithreaded test: see PR Request https://github.com/JuliaParallel/MPI.jl/pull/724
The draft PR is based off the most recent commit where all tests passed (Tag 0.20.8). In the output of "test-intel-linux", the salient output is
```
signal (11): Segmentation fault
in expression starting at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:18
ijl_gc_enable at /cache/build/default-amdci4-2/julialang/julia-release-1-dot-8/src/gc.c:2955
```
The change we made is in the file test/test_threads.jl, where we added the following if clause:
```
Threads.@threads for i = 1:N
reqs[N+i] = MPI.Irecv!(@view(recv_arr[i:i]), comm; source=src, tag=i)
reqs[i] = MPI.Isend(@view(send_arr[i:i]), comm; dest=dst, tag=i)
if i == 1
GC.gc()
end
end
```
We experience similar problems with MPICH 4.0 in our package (https://github.com/Julia-Tempering/Pigeons.jl), but not with MPICH 4.1.
**Related discussions**
- https://juliaparallel.org/MPI.jl/stable/knownissues/#Multi-threading-and-signal-handling
This describes a similar issue in the context of UCX. However this problem does not seem limited to UCX from our investigations so far.
- https://github.com/JuliaParallel/MPI.jl/issues/337
This describes a similar issue in the context of OpenMPI. However it seems that certain versions of MPICH and intel MPI (which is MPICH-derived) might suffer from a similar issue?
In light of these two sources, perhaps other environment variables in the style of https://github.com/JuliaParallel/MPI.jl/blob/6d513bb37da76182f2c31baff44f1f53a103295d/src/MPI.jl#L133 could be set to address this issue? I was wondering if anyone might have some suggestion on whether that's a reasonable hypothesis? Having limited MPI experience I am not sure what these environment variables might be.
Thank you so much for your time.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.