`pmap`'s `on_error` and `retry_delays` work differently if a worker exits

But in the docstring, it sounds like it's just a matter of "do you want to retry" or "do you want to handle it yourself" when choosing between the two (or using both).

For example,

julia> using Distributed

julia> addprocs(4)
4-element Vector{Int64}:
 2
 3
 4
 5

julia> pmap(1:5; on_error=identity) do i
           myid() == 3 && error("no")
           i
       end
5-element Vector{Any}:
 1
 2
  ErrorException("no")
 4
 5

works as expected. But if instead of throwing an error, we exit,

julia> pmap(1:5; on_error=identity) do i
           myid() == 2 && exit(1)
           i
       end
Worker 2 terminated.
ERROR: ProcessExitedException(2)
Stacktrace:
 [1] (::Base.var"#892#894")(x::Task)
   @ Base ./asyncmap.jl:177
 [2] foreach(f::Base.var"#892#894", itr::Vector{Any})
   @ Base ./abstractarray.jl:2694
 [3] maptwice(wrapped_f::Function, chnl::Channel{Any}, worker_tasks::Vector{Any}, c::UnitRange{Int64})
   @ Base ./asyncmap.jl:177
 [4] wrap_n_exec_twice
   @ ./asyncmap.jl:153 [inlined]
 [5] #async_usemap#877
   @ ./asyncmap.jl:103 [inlined]
 [6] #asyncmap#876
   @ ./asyncmap.jl:81 [inlined]
 [7] pmap(f::Function, p::WorkerPool, c::UnitRange{Int64}; distributed::Bool, batch_size::Int64, on_error::Function, retry_delays::Vector{Any}, retry_check::Nothing)
   @ Distributed ~/.julia/juliaup/julia-1.7.2+0~x64/share/julia/stdlib/v1.7/Distributed/src/pmap.jl:126
 [8] pmap(f::Function, c::UnitRange{Int64}; kwargs::Base.Pairs{Symbol, typeof(identity), Tuple{Symbol}, NamedTuple{(:on_error,), Tuple{typeof(identity)}}})
   @ Distributed ~/.julia/juliaup/julia-1.7.2+0~x64/share/julia/stdlib/v1.7/Distributed/src/pmap.jl:156
 [9] top-level scope
   @ REPL[4]:1

julia> Unhandled Task ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#645")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:892
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:900
 [3] unsafe_read
   @ ./io.jl:724 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:723
 [5] read!
   @ ./io.jl:725 [inlined]
 [6] deserialize_hdr_raw
   @ ~/.julia/juliaup/julia-1.7.2+0~x64/share/julia/stdlib/v1.7/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed ~/.julia/juliaup/julia-1.7.2+0~x64/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:165
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed ~/.julia/juliaup/julia-1.7.2+0~x64/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:126
 [9] (::Distributed.var"#99#100"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:423

Then it fails and does not return the ProcessExitedException.

However, if we use retry_delays instead:

julia> pmap(1:5, retry_delays = ExponentialBackOff(n = 3)) do i
           myid() == 4 && exit(1)
           i
       end
Worker 4 terminated.
Unhandled Task ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#645")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:892
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:900
 [3] unsafe_read
   @ ./io.jl:724 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:723
 [5] read!
   @ ./io.jl:725 [inlined]
 [6] deserialize_hdr_raw
   @ ~/.julia/juliaup/julia-1.7.2+0~x64/share/julia/stdlib/v1.7/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed ~/.julia/juliaup/julia-1.7.2+0~x64/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:165
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed ~/.julia/juliaup/julia-1.7.2+0~x64/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:126
 [9] (::Distributed.var"#99#100"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:423
5-element Vector{Int64}:
 1
 2
 3
 4
 5

Then it works fine and handles the issue (note it returns the result; the exception is printed but doesn't stop execution).

I am glad that retry_delays work to handle the case of a worker exiting (...which is happening more than I'd like it to in some code I'm running), but it would be great if on_error could also handle it, or if the docstring could be more clear about the differences between the two.

I suspect this is a known issue, but I couldn't find an existing issue for it.

For completeness, if you pass both, then (...please wait while I add some more workers...) you get

julia> pmap(1:5; on_error=identity, retry_delays=ExponentialBackOff(n = 3)) do i
           myid() == 6 && exit(1)
           i
       end
Worker 6 terminated.
Unhandled Task ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#645")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:892
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:900
 [3] unsafe_read
   @ ./io.jl:724 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:723
 [5] read!
   @ ./io.jl:725 [inlined]
 [6] deserialize_hdr_raw
   @ ~/.julia/juliaup/julia-1.7.2+0~x64/share/julia/stdlib/v1.7/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed ~/.julia/juliaup/julia-1.7.2+0~x64/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:165
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed ~/.julia/juliaup/julia-1.7.2+0~x64/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:126
 [9] (::Distributed.var"#99#100"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:423
5-element Vector{Int64}:
 1
 2
 3
 4
 5

Start with Distributed/src/pmap.jl, especially the pmap path shown in the stack trace, and compare how on_error and retry_delays handle a worker exit. The related asyncmap.jl and process_messages.jl traces show surrounding failure handling. Done means worker termination is handled consistently or the pmap docstring clearly explains the difference, with coverage for the shown cases.