JuliaParallel / JuliaParallel/Dagger.jl

Scheduler seems too eager on pushing jobs to workers

Open
#286 13 comments 0 reactions 0 assignees View on GitHub
bug scheduler
Dominant language
Julia
Stars
723
Forks
90
Avg merge
1d 37m
Merged PRs (30d)
9

Description

Main problem is that adding more workers to the pool is ineffective since there are no unassigned tasks.

I suppose this might be done to reduce scheduling latency. If it is, it would be useful to be able to control the amount of overprovisioning for cases when one intend to add more workers (e.g. as jobs are started on a cluster).

```julia
julia> using Distributed, Dagger

julia> addprocs(5, exeflags="--project=test")
5-element Vector{Int64}:
2
3
4
5
6

julia> @everywhere begin using Dagger, Distributed
function expensivejob(i)
@info "Started task $i on $(myid())"
while true end
end
end
julia> ts = delayed(vcat)([delayed(expensivejob)(i) for i in 1:200]...);

julia> collect(ts)
[ Info: Started task 2 on 3
[ Info: Started task 7 on 3
[ Info: Started task 12 on 3
[ Info: Started task 17 on 3
[ Info: Started task 22 on 3
[ Info: Started task 27 on 3
...
[ Info: Started task 180 on 6
[ Info: Started task 185 on 6
[ Info: Started task 190 on 6
[ Info: Started task 195 on 6
[ Info: Started task 200 on 6
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.