Large time delay/white space between subsequent dask.compute() calls
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**What happened**:
I use dask as a way to do parallel processing of single functions, and [the repo where I use it](https://github.com/scottshambaugh/monaco/blob/main/src/monaco/mc_sim.py) essentially runs three different single-item task graphs one after the other. However, I am seeing huge time delays between consecutive runs of the task graphs.
I have tried to boil down the issue to a minimum viable example which should have essentially no delay due to serialization or surrounding code. The below example mirrors the rough structure of how I use dask in my code, and reproduces the the issue I am seeing. Attached is the html performance report, but in short this the large delay between the two groups of tasks is what I'm looking at:

I'm happy to do some more digging into what could be happening, but I have no idea how to interrogate what is happening in that gap.
**What you expected to happen**:
There should not be such a large delay between the two groups of tasks.
**Minimal Complete Verifiable Example**:
```python
import dask
from dask.distributed import Client, performance_report
from numpy.random import default_rng
def case_runner(case):
return round(case)
class Sim:
def __init__(self, ncases=int(1e3)):
self.ncases = ncases
self.rng = default_rng(seed=12345)
self.cases = self.rng.random(self.ncases)
self.runcases = []
self.client = Client()
self.cluster = self.client.cluster
print(f'Dask dashboard link: {self.cluster.dashboard_link}')
def run(self, n):
with performance_report():
for _ in range(n):
runcases = []
for case in self.cases:
case_delayed = dask.delayed(case_runner)(case)
runcases.append(case_delayed)
self.runcases = dask.compute(*runcases)
def main():
sim = Sim(ncases=int(1e3))
sim.run(n=2)
return
if __name__ == '__main__':
main()
```
**Anything else we need to know?**:
**Environment**:
Note that this behavior is not new and has been showing itself since I started using dask at the start of this year.
- Dask version: 2022.9.1
- Python version: 3.10
- Operating System: Ubuntu running on Windows via WSL2
- Install method (conda, pip, source): pip
Cluster Dump State:
[dask-report.zip](https://github.com/dask/distributed/files/9641862/dask-report.zip)
Contributor guide
Assessment
This issue has not been assessed yet.