dask / dask/distributed

Dynamic adaption of average task duration

Open
#3,516 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

We currently experience an issue with very long-running tasks, when using `adaptive_target` to figure out how many workers the dask graph needs.
The estimation by `adaptive_target` is currently wrong as long as no task of the TaskPrefix has been executed yet.
Could we change the estimation to correct dynamically and propagate a new average across the cluster as soon as the execution of one task e.g. already takes twice the estimated time?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.