dask / dask/distributed

Performance report compute time stats are inaccurate with lots of tasks

Open
#5,401 3 comments 0 reactions 0 assignees View on GitHub
diagnostics good second issue
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

If there are more tasks than the task stream length limit (`distributed.scheduler.dashboard.tasks.task-stream-length`), then I think the calculation of "number of tasks", "compute time", etc. would be an underestimate, because the deque of tasks would roll over. When generating the performance report, we're just summing up data from the task stream: https://github.com/dask/distributed/blob/51a63ea8082f723d0e67411ab86b154d96eb0c43/distributed/scheduler.py#L7327-L7335

I'd propose that in `Scheduler.performance_report`, if `total_tasks == self.plugins[TaskStreamPlugin.name].buffer.maxlen` (or something like that, but written more nicely), we just prepend add a `>=` to every value in the performance report ("number of tasks: >=100000", "compute time: >= 123456s", etc.).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.