dask / dask/distributed

Large time delay/white space between subsequent dask.compute() calls

Open
#7,066 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**What happened**:
I use dask as a way to do parallel processing of single functions, and [the repo where I use it](https://github.com/scottshambaugh/monaco/blob/main/src/monaco/mc_sim.py) essentially runs three different single-item task graphs one after the other. However, I am seeing huge time delays between consecutive runs of the task graphs.

I have tried to boil down the issue to a minimum viable example which should have essentially no delay due to serialization or surrounding code. The below example mirrors the rough structure of how I use dask in my code, and reproduces the the issue I am seeing. Attached is the html performance report, but in short this the large delay between the two groups of tasks is what I'm looking at:

![image](https://user-images.githubusercontent.com/14363975/192167057-97f406a4-23b1-41f0-8ea9-be24a14757bc.png)

I'm happy to do some more digging into what could be happening, but I have no idea how to interrogate what is happening in that gap.

**What you expected to happen**:
There should not be such a large delay between the two groups of tasks.

**Minimal Complete Verifiable Example**:

```python
import dask
from dask.distributed import Client, performance_report
from numpy.random import default_rng

def case_runner(case):
return round(case)

class Sim:
def __init__(self, ncases=int(1e3)):
self.ncases = ncases
self.rng = default_rng(seed=12345)
self.cases = self.rng.random(self.ncases)
self.runcases = []

self.client = Client()
self.cluster = self.client.cluster
print(f'Dask dashboard link: {self.cluster.dashboard_link}')

def run(self, n):
with performance_report():
for _ in range(n):
runcases = []
for case in self.cases:
case_delayed = dask.delayed(case_runner)(case)
runcases.append(case_delayed)
self.runcases = dask.compute(*runcases)

def main():
sim = Sim(ncases=int(1e3))
sim.run(n=2)
return

if __name__ == '__main__':
main()
```

**Anything else we need to know?**:

**Environment**:
Note that this behavior is not new and has been showing itself since I started using dask at the start of this year.

- Dask version: 2022.9.1
- Python version: 3.10
- Operating System: Ubuntu running on Windows via WSL2
- Install method (conda, pip, source): pip

Cluster Dump State:

[dask-report.zip](https://github.com/dask/distributed/files/9641862/dask-report.zip)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.