JuliaLang / JuliaLang/Distributed.jl

rmprocs / addprocs racy

Open
#38 1 comment 0 reactions 1 assignee View on GitHub

@amitmurthy is already working on this.

Since Dec 23, 2016.

Dominant language
Julia
Stars
55
Forks
19
PR merge metrics
No merged PRs in 30d

Description

node termination during node provisioning is not well handled, resulting in `connect: connection refused (ECONNREFUSED) in connect_to_worker from the new worker to the terminating worker.

for an example, see: https://travis-ci.org/JuliaLang/julia/jobs/186141590

julia> p = addprocs(2)

julia> begin # try this a couple times
         @spawnat p[1] sleep(5)
         @show rmprocs(p[1]; waitfor=0)
         @show workers()
         @show p = addprocs(1)
       end
rmprocs(p[1]; waitfor=0) = :ok
workers() = [3,4,5]
ERROR: connect: connection refused (ECONNREFUSED)
 in yieldto(::Task, ::ANY) at ./event.jl:153
 in wait() at ./event.jl:186
 in wait(::Condition) at ./event.jl:27
 in stream_wait(::TCPSocket, ::Condition, ::Vararg{Condition,N}) at ./stream.jl:42
 in wait_connected(::TCPSocket) at ./stream.jl:258
 in connect at ./stream.jl:957 [inlined]
 in connect_to_worker(::String, ::Int16) at ./managers.jl:490
 in connect_w2w(::Int64, ::WorkerConfig) at ./managers.jl:453
 in connect(::Base.DefaultClusterManager, ::Int64, ::WorkerConfig) at ./managers.jl:387
 in connect_to_peer(::Base.DefaultClusterManager, ::Int64, ::WorkerConfig) at ./multi.jl:1516
 in (::Base.##598#600{WorkerConfig,Int64})() at ./task.jl:404
Error [connect: connection refused (ECONNREFUSED)] on 6 while connecting to peer 4. Exiting.
Worker 6 terminated.
ERROR (unhandled task failure): Version read failed. Connection closed by peer.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.