JuliaParallel / JuliaParallel/Dagger.jl

Task cost estimation: Treat non-Chunk inputs as residing on scheduler

Open
#330 0 comments 0 reactions 0 assignees View on GitHub
data movement performance scheduler
Dominant language
Julia
Stars
723
Forks
90
Avg merge
1d 37m
Merged PRs (30d)
9

Description

We currently ignore the costs of transferring non-`Chunk` task inputs, but that's silly; they have a real transfer cost, but it's incurred immediately at task transfer to the worker. We should calculate the sizes of these inputs (which we can also cache locally) and factor them into the costs in `estimate_task_costs`. This should cause the scheduler to bias tasks with heavy non-`Chunk` inputs to the scheduler's worker (but on a different thread).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.