Dynamic adaption of average task duration
Open
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
We currently experience an issue with very long-running tasks, when using `adaptive_target` to figure out how many workers the dask graph needs.
The estimation by `adaptive_target` is currently wrong as long as no task of the TaskPrefix has been executed yet.
Could we change the estimation to correct dynamically and propagate a new average across the cluster as soon as the execution of one task e.g. already takes twice the estimated time?
Contributor guide
Assessment
This issue has not been assessed yet.