PyCall crashes remote workers on 0.5
- Lingua principale
- Julia
- Stelle
- 1.5k
- Fork
- 186
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
Having an issue where calling `@everywhere using PyCall` crashes out remote workers.
I had this issue before:
- deleted the PyCall directory,
- recloned the git repo in my `~/.julia/v0.5` folder
- ran `Pkg.build("PyCall")`
- used `@everywhere using PyCall` and it worked
- exited later due to some other issue after using some @pyimported stuff successfully
- restarted Julia
- ran into the issue again, hence the post here.
Running `using PyCall` on the master process doesn't produce an error.
Here's a dump of what went on:
```
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type "?help" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.5.0 (2016-09-19 18:14 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org/ release
|__/ | x86_64-pc-linux-gnu
julia> # SETUP THE CLUSTER
using ClusterManagers
julia> using ClusterUtils
WARNING: deprecated syntax "[a=>b for (a,b) in c]".
Use "Dict(a=>b for (a,b) in c)" instead.
WARNING: Method definition del_client(Any, Any, Any) in module Base at multi.jl:865 overwritten in module ClusterUtils at /home/mcp50/.julia/v0.5/ClusterUtils/src/ClusterUtils.jl:366.
julia> numnodes = 2
2
julia> nodecore = 1
1
julia> numprocs = numnodes*nodecore
2
julia> remotes = addprocs(SlurmManager(numprocs), partition="mrc-bsu-tesla", account="MRC-BSU-SL2-GPU", t="24:00:00", nodes=numnodes);
srun: job 3711471 queued and waiting for resources
srun: job 3711471 has been allocated resources
connecting to worker 2 out of 2
julia> # ENSURE WE USE MULTITHREADING
if VERSION @everywhere FFTW.set_num_threads(12)
julia> @everywhere using PyCall
Worker 2 terminated.
Worker 3 terminated.ERROR (unhandled task failure): EOFError: read end of file
ERROR: ProcessExitedException()
in yieldto(::Task, ::ANY) at ./event.jl:136
in wait() at ./event.jl:169
in wait(::Condition) at ./event.jl:27
in wait(::Channel{Any}) at ./channels.jl:92
in take!(::Channel{Any}) at ./channels.jl:73
in #remotecall_fetch#606(::Array{Any,1}, ::Function, ::Function, ::Base.Worker) at ./multi.jl:1066
in remotecall_fetch(::Function, ::Base.Worker) at ./multi.jl:1062
in #remotecall_fetch#609(::Array{Any,1}, ::Function, ::Function, ::Int64) at ./multi.jl:1080
in remotecall_fetch(::Function, ::Int64) at ./multi.jl:1080
in (::##14#16)() at ./multi.jl:1959
...and 2 other exceptions.
in sync_end() at ./task.jl:311
in macro expansion; at ./multi.jl:1968 [inlined]
in anonymous at ./:?
ERROR (unhandled task failure): EOFError: read end of file
julia> srun: error: mrc-bsu-tesla2: task 1: Segmentation fault
srun: error: mrc-bsu-tesla1: task 0: Segmentation fault
```
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Start by reproducing the reported Julia 0.5 cluster setup with ClusterManagers, ClusterUtils, SlurmManager, and two workers, comparing `using PyCall` on the master with `@everywhere using PyCall`. Use the worker termination and Slurm segmentation-fault output as the initial failure signal; done means remote workers remain available after PyCall is loaded.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- distributed-systems
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Ferma
- Chiarezza
- Da chiarire
- Idoneità per principianti
- 25/100