invenia / invenia/Parallelism.jl
Performance hazard in `fetch()`ing from a Task
- Dominant language
- Julia
- Stars
- 8
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
I just discovered this repo today, and did some scanning through it, and encountered this:
https://github.com/invenia/Parallelism.jl/blob/159a138d562a1583e44129340a0dfa784a34c523/src/tmap.jl#L9-L12
This will end up boxing every value as an Any, so you'll end up always returning a `Vector{Any}`.
Probably this would be better served by pre-allocating a vector of `T`s, and then assigning the results into it? But I guess you don't know what the types will be.... Maybe you could either take it as an input to the function, or use `Core.Compiler.return_type()` like Julia does?:
https://github.com/JuliaLang/julia/blob/55a6dab76329b693f0fab372b1a80289bff01a90/base/array.jl#L660
So ultimately, something like:
```julia
T = ... # Maybe `Core.Compiler.return_type(f, eltype(xss))`?
vs = Vector{T}(undef, length(xss))
for (i,x) in enumerate(xss)
Threads.@spawn vs[i] = f(x...)
end
return vs
```
This is the pattern we've been using so far, and it seems to mostly work okay.
Contributor guide
No contributing guide indexed for this repository
Research direction
Read src/tmap.jl lines 9-12 first, then compare its Task-fetching pattern with Julia's base array implementation and the suggested Core.Compiler.return_type approach. Done means the parallel mapping result no longer defaults to Vector{Any} while preserving the existing threaded behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100