JuliaParallel / JuliaParallel/MPI.jl

test_spawn.jl fails (tested on Cray system with Cray mpt 7.7.10)

Open
#385 1 comment 0 reactions 0 assignees View on GitHub
Cray MPI
Dominant language
Julia
Stars
439
Forks
127
Avg merge
19h 6m
Merged PRs (30d)
17

Description

That's maybe because as one cannot spawn whatever you want when allocated with SLURM.

```
srun: Warning: can't run 1 processes on 4 nodes, setting nnodes to 1
Rank 0 [Fri May 8 18:08:48 2020] [c0-0c0s0n0] Fatal error in MPI_Comm_spawn: Other MPI error, error stack:
MPI_Comm_spawn(144).........: MPI_Comm_spawn(cmd="/scratch/snx3000tds/omlins/9_soft_install_julia_1_4/software/Julia/1.4.1-CrayGNU-19.10-cuda-10.1/bin/julia", argv=0x2aaabada7290, maxprocs=3, MPI_INFO_NULL, root=0, MPI_COMM_WORLD, intercomm=0x2aaabb379050, errors=0x2aaaba2c2f90) faile
MPID_Comm_spawn_multiple(70): Function MPID_Comm_spawn_multiple not implemented

signal (6): Aborted
in expression starting at /users/omlins/.julia/1.4.1/dom-gpu/dev/MPI/test/test_spawn.jl:11
gsignal at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
MPID_Abort at /opt/cray/pe/mpt/7.7.10/gni/mpich-gnu/8.2/lib/libmpich.so (unknown line)
MPIR_Handle_fatal_error at /opt/cray/pe/mpt/7.7.10/gni/mpich-gnu/8.2/lib/libmpich.so (unknown line)
MPIR_Err_return_comm at /opt/cray/pe/mpt/7.7.10/gni/mpich-gnu/8.2/lib/libmpich.so (unknown line)
PMPI_Comm_spawn at /opt/cray/pe/mpt/7.7.10/gni/mpich-gnu/8.2/lib/libmpich.so (unknown line)
#Comm_spawn#25 at /users/omlins/.julia/1.4.1/dom-gpu/dev/MPI/src/comm.jl:141
Comm_spawn at /users/omlins/.julia/1.4.1/dom-gpu/dev/MPI/src/comm.jl:137
unknown function (ip: 0x2aaabf21af16)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2158 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:369
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:458
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:409 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:817
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:911
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:814
jl_parse_eval_all at /buildworker/worker/package_linux64/build/src/ast.c:872
jl_load at /buildworker/worker/package_linux64/build/src/toplevel.c:872
include at ./Base.jl:377
exec_options at ./client.jl:288
_start at ./client.jl:484
jfptr__start_2075.clone_1 at /scratch/snx3000tds/omlins/9_soft_install_julia_1_4/software/Julia/1.4.1-CrayGNU-19.10-cuda-10.1/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/ui/../src/julia.h:1700 [inlined]
true_main at /buildworker/worker/package_linux64/build/ui/repl.c:96
main at /buildworker/worker/package_linux64/build/ui/repl.c:217
__libc_start_main at /lib64/libc.so.6 (unknown line)
_start at /scratch/snx3000tds/omlins/9_soft_install_julia_1_4/software/Julia/1.4.1-CrayGNU-19.10-cuda-10.1/bin/julia (unknown line)
Allocations: 1701693 (Pool: 1701268; Big: 425); GC: 2
srun: error: nid00000: task 0: Aborted (core dumped)
srun: Terminating job step 1015464.25
ERROR: LoadError: failed process: Process(`srun -C gpu -n 1 /scratch/snx3000tds/omlins/9_soft_install_julia_1_4/software/Julia/1.4.1-CrayGNU-19.10-cuda-10.1/bin/julia --code-coverage=none /users/omlins/.julia/1.4.1/dom-gpu/dev/MPI/test/test_spawn.jl`, ProcessExited(134)) [134]

Stacktrace:
[1] pipeline_error at ./process.jl:525 [inlined]
[2] run(::Cmd; wait::Bool) at ./process.jl:440
[3] run at ./process.jl:438 [inlined]
[4] (::var"#3#7"{Int64,String,String,String,String})(::Cmd) at /users/omlins/.julia/1.4.1/dom-gpu/dev/MPI/test/runtests.jl:37
[5] mpiexec(::var"#3#7"{Int64,String,String,String,String}) at /users/omlins/.julia/1.4.1/dom-gpu/dev/MPI/deps/deps.jl:11
[6] runtests() at /users/omlins/.julia/1.4.1/dom-gpu/dev/MPI/test/runtests.jl:34
[7] top-level scope at /users/omlins/.julia/1.4.1/dom-gpu/dev/MPI/test/runtests.jl:53
[8] include(::Module, ::String) at ./Base.jl:377
[9] exec_options(::Base.JLOptions) at ./client.jl:288
[10] _start() at ./client.jl:484
in expression starting at /users/omlins/.julia/1.4.1/dom-gpu/dev/MPI/test/runtests.jl:53
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.