JuliaLang / JuliaLang/Distributed.jl

pmap error handling fails with batch_size > 1

Open
#162 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
55
Forks
19
PR merge metrics
No merged PRs in 30d

Description

Here is a MWE

julia> using Distributed

julia> addprocs(2)

julia> pmap(o->error(),zeros(2),on_error= ex->0.0,batch_size=1) # Works as expected
2-element Array{Float64,1}:
 0.0
 0.0

julia> pmap(o->error(),zeros(2),on_error= ex->0.0,batch_size=2) # Fails
ERROR: MethodError: Cannot `convert` an object of type Float64 to an object of type Distributed.BatchProcessingError
Closest candidates are:
  convert(::Type{T}, ::T) where T at essentials.jl:154
  Distributed.BatchProcessingError(::Any, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:4
Stacktrace:
 [1] setindex!(::Array{Distributed.BatchProcessingError,1}, ::Float64, ::Int64) at ./array.jl:767
 [2] process_batch_errors!(::WorkerPool, ::Function, ::Array{Distributed.BatchProcessingError,1}, ::Function, ::Array{Any,1}, ::Nothing) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:228
 [3] #pmap#213(::Bool, ::Int64, ::Function, ::Array{Any,1}, ::Nothing, ::Function, ::Function, ::WorkerPool, ::Array{Float64,1}) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:148
 [4] (::getfield(Distributed, Symbol("#kw##pmap")))(::NamedTuple{(:on_error, :batch_size),Tuple{getfield(Main, Symbol("##44#46")),Int64}}, ::typeof(pmap), ::Function, ::WorkerPool, ::Array{Float64,1}) at ./none:0
 [5] #pmap#223(::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:on_error, :batch_size),Tuple{getfield(Main, Symbol("##44#46")),Int64}}}, ::Function, ::Function, ::Array{Float64,1}) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:156
 [6] (::getfield(Distributed, Symbol("#kw##pmap")))(::NamedTuple{(:on_error, :batch_size),Tuple{getfield(Main, Symbol("##44#46")),Int64}}, ::typeof(pmap), ::Function, ::Array{Float64,1}) at ./none:0
 [7] top-level scope at none:0
julia> versioninfo()
Julia Version 1.1.1
Commit 55e36cc (2019-05-16 04:10 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin15.6.0)
  CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 12
  JULIA_EDITOR = nvim

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the Julia 1.1.1 MWE with Distributed.pmap and compare batch_size=1 with batch_size=2. Start in Distributed/src/pmap.jl at process_batch_errors!, identified in the stack trace; done means on_error returns 0.0 successfully for batch_size=2 without a BatchProcessingError conversion failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
distributed-systems
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.