JuliaPy / JuliaPy/pyjulia

pyjulia segfaults with multiprocessing

Open
#362 9 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
893
Forks
101
PR merge metrics
No merged PRs in 30d

Description

Calling Julia code using Python's multiprocessing module crashes the program (SIGSEGV). I am using Julia 1.2.0, Python 3.7.3 and pyjulia 0.4.1 on Linux.

A minimal example is as follows:
```
import multiprocessing as mp
import julia

def test(*args):
j = julia.Julia()

myfunc = j.eval("function test() return 5; end")
myfunc()

with mp.Pool(10) as p:
p.map(test, range(10))
```

This program crashes with a high probability which increases with the number of children. Based on the actual code I need to run it seems like the crash occurs whenever a Julia-defined function is called for the first time in a process; if this succeeds the program does not seem to crash even after many iterations.

```
Warning: `getindex(o::PyObject, s::Symbol)` is deprecated in favor of dot overloading (`getproperty`) so elements should now be accessed as e.g. `o.s` instead of `o[:s]`.
│ caller = top-level scope at none:4
└ @ Core none:4

signal (11): Segmentation fault
in expression starting at none:1
unknown function (ip: 0xffffffffffffffff)
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2197
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1614 [inlined]
call_require at /buildworker/worker/package_linux64/build/src/toplevel.c:399 [inlined]
eval_import_path at /buildworker/worker/package_linux64/build/src/toplevel.c:436
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:656
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:764
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:844
jl_eval_string at /buildworker/worker/package_linux64/build/src/jlapi.c:94
ffi_call_unix64 at /home/user/.miniconda3/lib/python3.7/lib-dynload/../../libffi.so.6 (unknown line)
ffi_call at /home/user/.miniconda3/lib/python3.7/lib-dynload/../../libffi.so.6 (unknown line)
_call_function_pointer at /usr/local/src/conda/python-3.7.3/Modules/_ctypes/callproc.c:827 [inlined]
_ctypes_callproc at /usr/local/src/conda/python-3.7.3/Modules/_ctypes/callproc.c:1184
PyCFuncPtr_call at /usr/local/src/conda/python-3.7.3/Modules/_ctypes/_ctypes.c:3969
_PyObject_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_Call_Prepend at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
slot_tp_init at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
type_call.lto_priv.151 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
PyObject_Call at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_Call_Prepend at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
slot_tp_init at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
type_call.lto_priv.151 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
map_next.lto_priv.527 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
list_extend.lto_priv.888 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
list___init__.lto_priv.379 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
type_call.lto_priv.151 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_Call_Prepend at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
slot_tp_init at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
type_call.lto_priv.151 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_Call_Prepend at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
slot_tp_init at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
type_call.lto_priv.151 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
PyEval_EvalCodeEx at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
PyEval_EvalCode at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
builtin_exec at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyMethodDef_RawFastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyCFunction_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
macro expansion at /home/user/.julia/packages/PyCall/ttONZ/src/exception.jl:81 [inlined]
__pycall! at /home/user/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:44
_pycall! at /home/user/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:29
#call#111 at /home/user/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:11 [inlined]
PyObject at /home/user/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:89
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2197
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:323
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:411
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:635
jl_interpret_toplevel_thunk_callback at /buildworker/worker/package_linux64/build/src/interpreter.c:884
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7f67cc984f0f)
unknown function (ip: 0x2)
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:893
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:815
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:764
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:844
eval at ./boot.jl:330
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2191
exec_options at ./client.jl:271
_start at ./client.jl:464
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2191
unknown function (ip: 0x40192d)
unknown function (ip: 0x401533)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4015d4)
Allocations: 11458870 (Pool: 11456484; Big: 2386); GC: 25
^C^C^C^C^C^C^CWARNING: Force throwing a SIGINT

signal (11): Segmentation fault
in expression starting at none:4
tupledealloc.lto_priv.484 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
pydecref_ at /home/user/.julia/packages/PyCall/ttONZ/src/PyCall.jl:121
_pycall! at /home/user/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:31
#call#111 at /home/user/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:11 [inlined]
PyObject at /home/user/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:89
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2197
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:323
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:411
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:635
jl_interpret_toplevel_thunk_callback at /buildworker/worker/package_linux64/build/src/interpreter.c:884
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7f67cc984f0f)
unknown function (ip: 0x2)
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:893
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:815
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:764
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:844
^Ceval at ./boot.jl:330
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2191
exec_options at ./client.jl:271
_start at ./client.jl:464
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2191
unknown function (ip: 0x40192d)^C
unknown function (ip: 0x401533)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4015d4)
Allocations: 8918253 (Pool: 8916828; Big: 1425); GC: 19
/bin/bash: line 1: 24040 Segmentation fault
```

This seems related to #344 and #306 (where I recently remarked the issue). Using `python-jl` does not seem to make a difference.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.