JuliaParallel / JuliaParallel/Dagger.jl
Task cost estimation: Treat non-Chunk inputs as residing on scheduler
Open
data movement
performance
scheduler
- Dominant language
- Julia
- Stars
- 723
- Forks
- 90
- Avg merge
- 1d 37m
- Merged PRs (30d)
- 9
Description
We currently ignore the costs of transferring non-`Chunk` task inputs, but that's silly; they have a real transfer cost, but it's incurred immediately at task transfer to the worker. We should calculate the sizes of these inputs (which we can also cache locally) and factor them into the costs in `estimate_task_costs`. This should cause the scheduler to bias tasks with heavy non-`Chunk` inputs to the scheduler's worker (but on a different thread).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.