dask / dask/distributed

Cost per task grows with number of tasks

Open
#2,611 10 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I noticed recently that the running time of my application grows faster than `O(N)` in the number of steps. This is an application written to issue tasks with `dask.delayed`. To quantify this I made a stripped down version that has the same dependence pattern with empty tasks, so all the time should be spend in the construction of the task graph.

I'm running a distributed setup with 1 scheduler and 1 worker on a machine with 32 cores per node. The worker has `--nthreads 32 --nprocs 1` but I don't think this matters since all the time is spent in graph construction. Running `htop` on the node shows the client is using 100% of one CPU core and the rest of the system is idle (and well below its memory limit).

My results are below. Time per task seems to grow with `O(N)` in the number of tasks. This means that total running time grows `O(N^2)`.

I realize that as a workaround I can probably call `compute()` more frequently---based on the data below it looks like about every 10 steps would be optimal. But I still want to check if this is a known performance behavior of Dask and whether it's worth looking at the application more closely to see if I might be doing something wrong.

Thanks in advance and let me know if this would be better submitted under another Dask subproject.

| Steps | Tasks | Total Time (s) | Time per Task (ms) |
|-------|-------|----------------|--------------------|
| 1 | 32 | 0.5316095 | 16.61279688 |
| 10 | 320 | 0.6279482 | 1.962338125 |
| 50 | 1600 | 7.029356 | 4.3933475 |
| 100 | 3200 | 25.81444 | 8.0670125 |
| 200 | 6400 | 107.7474 | 16.83553125 |
| 300 | 9600 | 236.0527 | 24.58882292 |

```
$ conda list
dask 1.1.5 py_0
dask-core 1.1.5 py_0
distributed 1.26.1 py37_0
python 3.7.1 h0371630_7
tornado 6.0.2 py37h7b6447c_0
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.