timholy / timholy/ProgressMeter.jl
Conflict with for-loop and explicit return in pmap
Open
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 789
- Forks
- 91
- PR merge metrics
- No merged PRs in 30d
Description
I've run into a strange edge case with @showprogress and pmap. First I import packages and add a worker
julia> using ProgressMeter
julia> using Distributed
julia> addprocs(1)
1-element Vector{Int64}:
2
Then I try running this:
julia> @showprogress for i = 1:10
pmap(1:10) do j
return j
end
sleep(0.1)
end
Which fails with this very long error about not being able to find ProgressMeter:
First error message
julia> @showprogress for i = 1:10
pmap(1:10) do j
return j
end
sleep(0.1)
end
ERROR: On worker 2:
KeyError: key ProgressMeter [92933f4c-e287-5a05-a399-4b506db050ca] not found
Stacktrace:
[1] getindex
@ ./dict.jl:498 [inlined]
[2] macro expansion
@ ./lock.jl:223 [inlined]
[3] root_module
@ ./loading.jl:1254
[4] deserialize_module
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:993
[5] handle_deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:895
[6] deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813
[7] deserialize_datatype
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1363
[8] handle_deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:866
[9] deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813
[10] handle_deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:873
[11] deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813
[12] deserialize_expr
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1266
[13] handle_deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:893
[14] deserialize_fillarray!
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1256
[15] deserialize_array
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1248
[16] handle_deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:864
[17] deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813 [inlined]
[18] deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1129
[19] handle_deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:959
[20] deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813 [inlined]
[21] deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1040
[22] handle_deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:959
[23] deserialize_fillarray!
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1256
[24] deserialize_array
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1248
[25] handle_deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:864
[26] deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813 [inlined]
[27] deserialize_typename
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:1326
[28] deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/clusterserialize.jl:68
[29] handle_deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:959
[30] deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813
[31] handle_deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:870
[32] deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813
[33] handle_deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:873
[34] deserialize
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Serialization/src/Serialization.jl:813 [inlined]
[35] deserialize_msg
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/messages.jl:87
[36] #invokelatest#2
@ ./essentials.jl:729 [inlined]
[37] invokelatest
@ ./essentials.jl:726 [inlined]
[38] message_handler_loop
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:176
[39] process_tcp_streams
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
[40] #103
@ ./task.jl:484
Stacktrace:
[1] (::Base.var"#938#940")(x::Task)
@ Base ./asyncmap.jl:177
[2] foreach(f::Base.var"#938#940", itr::Vector{Any})
@ Base ./abstractarray.jl:2774
[3] maptwice(wrapped_f::Function, chnl::Channel{Any}, worker_tasks::Vector{Any}, c::UnitRange{Int64})
@ Base ./asyncmap.jl:177
[4] wrap_n_exec_twice
@ ./asyncmap.jl:153 [inlined]
[5] #async_usemap#923
@ ./asyncmap.jl:103 [inlined]
[6] #asyncmap#922
@ ./asyncmap.jl:81 [inlined]
[7] pmap(f::Function, p::WorkerPool, c::UnitRange{Int64}; distributed::Bool, batch_size::Int64, on_error::Nothing, retry_delays::Vector{Any}, retry_check::Nothing)
@ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:126
[8] pmap(f::Function, p::WorkerPool, c::UnitRange{Int64})
@ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:99
[9] pmap(f::Function, c::UnitRange{Int64}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:156
[10] pmap(f::Function, c::UnitRange{Int64})
@ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:156
[11] macro expansion
@ REPL[10]:2 [inlined]
[12] top-level scope
@ ~/.julia/packages/ProgressMeter/sN2xr/src/ProgressMeter.jl:938
Then I import ProgressMeter everywhere and try again:
julia> @everywhere using ProgressMeter
julia> @showprogress for i = 1:10
pmap(1:10) do j
return j
end
sleep(0.1)
end
This fails with a different error:
Second error message
julia> @showprogress for i = 1:10
pmap(1:10) do j
return j
end
sleep(0.1)
end
ERROR: On worker 2:
UndefVarError: ##meter#313 not defined
Stacktrace:
[1] macro expansion
@ ~/.julia/packages/ProgressMeter/sN2xr/src/ProgressMeter.jl:710 [inlined]
[2] #3
@ ./REPL[17]:3
[3] #invokelatest#2
@ ./essentials.jl:729
[4] invokelatest
@ ./essentials.jl:726
[5] #110
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285
[6] run_work_thunk
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:70
[7] macro expansion
@ ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:285 [inlined]
[8] #109
@ ./task.jl:484
Stacktrace:
[1] (::Base.var"#938#940")(x::Task)
@ Base ./asyncmap.jl:177
[2] foreach(f::Base.var"#938#940", itr::Vector{Any})
@ Base ./abstractarray.jl:2774
[3] maptwice(wrapped_f::Function, chnl::Channel{Any}, worker_tasks::Vector{Any}, c::UnitRange{Int64})
@ Base ./asyncmap.jl:177
[4] wrap_n_exec_twice
@ ./asyncmap.jl:153 [inlined]
[5] #async_usemap#923
@ ./asyncmap.jl:103 [inlined]
[6] #asyncmap#922
@ ./asyncmap.jl:81 [inlined]
[7] pmap(f::Function, p::WorkerPool, c::UnitRange{Int64}; distributed::Bool, batch_size::Int64, on_error::Nothing, retry_delays::Vector{Any}, retry_check::Nothing)
@ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:126
[8] pmap(f::Function, p::WorkerPool, c::UnitRange{Int64})
@ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:99
[9] pmap(f::Function, c::UnitRange{Int64}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:156
[10] pmap(f::Function, c::UnitRange{Int64})
@ Distributed ~/.julia/juliaup/julia-1.8.0-rc4+0.x64/share/julia/stdlib/v1.8/Distributed/src/pmap.jl:156
[11] macro expansion
@ REPL[17]:2 [inlined]
[12] top-level scope
@ ~/.julia/packages/ProgressMeter/sN2xr/src/ProgressMeter.jl:938
But this works fine, even without @everywhere using ProgressMeter:
julia> @showprogress for i = 1:10
pmap(1:10) do j
j # There is no explicit return here
end
sleep(0.1)
end
For some reason the omission of return in the pmap body makes it work. I had a quick look through the source to see if I could find the problem, and I think it's something to do with showprogress_process_expr(): https://github.com/timholy/ProgressMeter.jl/blob/v1.7.2/src/ProgressMeter.jl#L707
As far as I can understand it, the code is recursively going through the loop body, and in this case where it finds a break/return it inserts a special node to handle break and return statements. The problem is that when there's a return in some code that's technically part of the loop but isn't actually executed in the loop (e.g. a closure), the macro will still insert those special nodes. I guess that explains the second error, UndefVarError: ##meter#313 not defined; the meter#313 variable is probably metersym in the code, which was created on the master process and then inserted into the AST to be executed on the worker.
I saw that #146 has similar error messages, but I don't think this is related.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the Julia reproducer in the issue with ProgressMeter, Distributed, addprocs(1), @showprogress, and pmap. Trace the interaction between the progress macro and the explicit return in the pmap callback; done means the example runs without the worker-side ProgressMeter KeyError and preserves progress reporting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- distributed-systems, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100