JuliaLang / JuliaLang/Distributed.jl
pmap error handling fails with batch_size > 1
Open
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 55
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
Here is a MWE
julia> using Distributed
julia> addprocs(2)
julia> pmap(o->error(),zeros(2),on_error= ex->0.0,batch_size=1) # Works as expected
2-element Array{Float64,1}:
0.0
0.0
julia> pmap(o->error(),zeros(2),on_error= ex->0.0,batch_size=2) # Fails
ERROR: MethodError: Cannot `convert` an object of type Float64 to an object of type Distributed.BatchProcessingError
Closest candidates are:
convert(::Type{T}, ::T) where T at essentials.jl:154
Distributed.BatchProcessingError(::Any, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:4
Stacktrace:
[1] setindex!(::Array{Distributed.BatchProcessingError,1}, ::Float64, ::Int64) at ./array.jl:767
[2] process_batch_errors!(::WorkerPool, ::Function, ::Array{Distributed.BatchProcessingError,1}, ::Function, ::Array{Any,1}, ::Nothing) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:228
[3] #pmap#213(::Bool, ::Int64, ::Function, ::Array{Any,1}, ::Nothing, ::Function, ::Function, ::WorkerPool, ::Array{Float64,1}) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:148
[4] (::getfield(Distributed, Symbol("#kw##pmap")))(::NamedTuple{(:on_error, :batch_size),Tuple{getfield(Main, Symbol("##44#46")),Int64}}, ::typeof(pmap), ::Function, ::WorkerPool, ::Array{Float64,1}) at ./none:0
[5] #pmap#223(::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:on_error, :batch_size),Tuple{getfield(Main, Symbol("##44#46")),Int64}}}, ::Function, ::Function, ::Array{Float64,1}) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:156
[6] (::getfield(Distributed, Symbol("#kw##pmap")))(::NamedTuple{(:on_error, :batch_size),Tuple{getfield(Main, Symbol("##44#46")),Int64}}, ::typeof(pmap), ::Function, ::Array{Float64,1}) at ./none:0
[7] top-level scope at none:0
julia> versioninfo()
Julia Version 1.1.1
Commit 55e36cc (2019-05-16 04:10 UTC)
Platform Info:
OS: macOS (x86_64-apple-darwin15.6.0)
CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
JULIA_NUM_THREADS = 12
JULIA_EDITOR = nvim
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the Julia 1.1.1 MWE with Distributed.pmap and compare batch_size=1 with batch_size=2. Start in Distributed/src/pmap.jl at process_batch_errors!, identified in the stack trace; done means on_error returns 0.0 successfully for batch_size=2 without a BatchProcessingError conversion failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100