JuliaParallel / JuliaParallel/MPI.jl

MPI implementations intercepting Signals is incompatible with Julia GC safepoint

Open
#725 15 comments 0 reactions 0 assignees View on GitHub
Intel MPI
Dominant language
Julia
Stars
439
Forks
127
Avg merge
19h 6m
Merged PRs (30d)
17

Description

Thanks again for your help with https://github.com/JuliaParallel/MPI.jl/issues/720 - this one is unrelated (except that issue #720 lead us to create more comprehensive unit test revealing this new, probably unrelated segfault).

**Summary of this problem:** a segfault occurs when GC is triggered in a multithreaded+MPI context.

**How to reproduce:** I have create a draft PR adding a GC.gc() call in one of MPI.jl's existing multithreaded test: see PR Request https://github.com/JuliaParallel/MPI.jl/pull/724

The draft PR is based off the most recent commit where all tests passed (Tag 0.20.8). In the output of "test-intel-linux", the salient output is

```
signal (11): Segmentation fault
in expression starting at /home/runner/work/MPI.jl/MPI.jl/test/test_threads.jl:18
ijl_gc_enable at /cache/build/default-amdci4-2/julialang/julia-release-1-dot-8/src/gc.c:2955
```

The change we made is in the file test/test_threads.jl, where we added the following if clause:

```
Threads.@threads for i = 1:N
reqs[N+i] = MPI.Irecv!(@view(recv_arr[i:i]), comm; source=src, tag=i)
reqs[i] = MPI.Isend(@view(send_arr[i:i]), comm; dest=dst, tag=i)
if i == 1
GC.gc()
end

end
```

We experience similar problems with MPICH 4.0 in our package (https://github.com/Julia-Tempering/Pigeons.jl), but not with MPICH 4.1.

**Related discussions**

- https://juliaparallel.org/MPI.jl/stable/knownissues/#Multi-threading-and-signal-handling

This describes a similar issue in the context of UCX. However this problem does not seem limited to UCX from our investigations so far.

- https://github.com/JuliaParallel/MPI.jl/issues/337

This describes a similar issue in the context of OpenMPI. However it seems that certain versions of MPICH and intel MPI (which is MPICH-derived) might suffer from a similar issue?

In light of these two sources, perhaps other environment variables in the style of https://github.com/JuliaParallel/MPI.jl/blob/6d513bb37da76182f2c31baff44f1f53a103295d/src/MPI.jl#L133 could be set to address this issue? I was wondering if anyone might have some suggestion on whether that's a reasonable hypothesis? Having limited MPI experience I am not sure what these environment variables might be.

Thank you so much for your time.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.