invenia / invenia/Parallelism.jl

Performance hazard in `fetch()`ing from a Task

Open
#7 3 comments 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Julia
Stars
8
Forks
1
PR merge metrics
No merged PRs in 30d

Description

I just discovered this repo today, and did some scanning through it, and encountered this:
https://github.com/invenia/Parallelism.jl/blob/159a138d562a1583e44129340a0dfa784a34c523/src/tmap.jl#L9-L12

This will end up boxing every value as an Any, so you'll end up always returning a `Vector{Any}`.

Probably this would be better served by pre-allocating a vector of `T`s, and then assigning the results into it? But I guess you don't know what the types will be.... Maybe you could either take it as an input to the function, or use `Core.Compiler.return_type()` like Julia does?:
https://github.com/JuliaLang/julia/blob/55a6dab76329b693f0fab372b1a80289bff01a90/base/array.jl#L660

So ultimately, something like:

```julia
T = ... # Maybe `Core.Compiler.return_type(f, eltype(xss))`?
vs = Vector{T}(undef, length(xss))
for (i,x) in enumerate(xss)
Threads.@spawn vs[i] = f(x...)
end
return vs
```
This is the pattern we've been using so far, and it seems to mostly work okay.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read src/tmap.jl lines 9-12 first, then compare its Task-fetching pattern with Julia's base array implementation and the suggested Core.Compiler.return_type approach. Done means the parallel mapping result no longer defaults to Vector{Any} while preserving the existing threaded behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.