JuliaPy / JuliaPy/PythonCall.jl

juliacall: Julia GC triggered from a python thread causes hangs

Ouverte
#578 3 commentaires 1 réaction 0 personnes assignées Voir sur GitHub
bug
Langage dominant
Julia
Étoiles
1.1k
Forks
86
Merge moyen
1 j 22 h
PR mergées (30 j)
3

Description

**Affects:** JuliaCall

**Describe the bug**
I am seeing Julia tasks launched from parallel python threads lead to sporadic hangs. Short-running tasks suffer less from this. I had no luck using the tricks from #539 or [here](https://juliapy.github.io/PythonCall.jl/stable/juliacall/#Caveat:-Julia's-task-scheduler). I think the hangs are triggered by Julia GC - disabling GC with `Base.GC.enable(false)` beforehand seems to fix the hangs - and triggering Julia GC manually from a python thread immediately causes a hang.

This is all with `PYTHON_JULIACALL_HANDLE_SIGNALS` set to `yes`, in case that matters.

```python
from juliacall import Main as jl
jl_gc = jl.Base.GC.gc

import threading
t = threading.Thread(target=jl_gc)
t.start() # hangs every time (100% CPU)
```

Terminating python gives me
```
[89363] signal (15): Terminated: 15
in expression starting at none:0
jl_gc_wait_for_the_world at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gc.c:241 [inlined]
ijl_gc_collect at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gc.c:3515
gc at ./gcutils.jl:129 [inlined]
gc at ./gcutils.jl:129 [inlined]
pyjlany_call at /Users/username/.julia/packages/PythonCall/Nr75f/src/JlWrap/any.jl:45
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gf.c:3077
(pyproj) c889f3b4c8d7:pyproj username$ _pyjl_callmethod at /Users/username/.julia/packages/PythonCall/Nr75f/src/JlWrap/base.jl:73
_pyjl_callmethod at /Users/username/.julia/packages/PythonCall/Nr75f/src/JlWrap/C.jl:63
jfptr__pyjl_callmethod_9032 at /Users/username/.julia/compiled/v1.10/PythonCall/WdXsa_v1O0R.dylib (unknown line)
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-R17H3W25T9.0/build/default-honeycrisp-R17H3W25T9-0/julialang/julia-release-1-dot-10/src/gf.c:3077
jlcapi__pyjl_callmethod_9116 at /Users/username/.julia/compiled/v1.10/PythonCall/WdXsa_v1O0R.dylib (unknown line)
method_vectorcall_VARARGS at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
PyObject_Vectorcall at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
_PyEval_EvalFrameDefault at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
_PyObject_FastCallDictTstate at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
slot_tp_call at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
_PyObject_Call at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
_PyEval_EvalFrameDefault at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
method_vectorcall at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
thread_run at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
pythread_wrapper at /Users/username/miniforge3/envs/pyproj/bin/python3.12 (unknown line)
_pthread_start at /usr/lib/system/libsystem_pthread.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
Allocations: 3689147 (Pool: 3684864; Big: 4283); GC: 6
```

**Your system**
- MacOS, Apple Silicon, Julia 1.10.5
- Linux, x86_64, Julia 1.10.6
- PythonCall/juliacall 0.9.23

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Start by running the supplied Python threading reproducer with JuliaCall and inspect the PythonCall JlWrap/any.jl and JlWrap/base.jl call paths shown in the stack trace. Compare behavior with Julia GC enabled and disabled; done means the threaded GC call completes without hanging on the reported environments.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
julia, python
Domaine
backend
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
À l'abandon
Clarté
À clarifier
Accessibilité débutants
30/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.