Performance report compute time stats are inaccurate with lots of tasks
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
If there are more tasks than the task stream length limit (`distributed.scheduler.dashboard.tasks.task-stream-length`), then I think the calculation of "number of tasks", "compute time", etc. would be an underestimate, because the deque of tasks would roll over. When generating the performance report, we're just summing up data from the task stream: https://github.com/dask/distributed/blob/51a63ea8082f723d0e67411ab86b154d96eb0c43/distributed/scheduler.py#L7327-L7335
I'd propose that in `Scheduler.performance_report`, if `total_tasks == self.plugins[TaskStreamPlugin.name].buffer.maxlen` (or something like that, but written more nicely), we just prepend add a `>=` to every value in the performance report ("number of tasks: >=100000", "compute time: >= 123456s", etc.).
Contributor guide
Assessment
This issue has not been assessed yet.