Cost per task grows with number of tasks
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I noticed recently that the running time of my application grows faster than `O(N)` in the number of steps. This is an application written to issue tasks with `dask.delayed`. To quantify this I made a stripped down version that has the same dependence pattern with empty tasks, so all the time should be spend in the construction of the task graph.
I'm running a distributed setup with 1 scheduler and 1 worker on a machine with 32 cores per node. The worker has `--nthreads 32 --nprocs 1` but I don't think this matters since all the time is spent in graph construction. Running `htop` on the node shows the client is using 100% of one CPU core and the rest of the system is idle (and well below its memory limit).
My results are below. Time per task seems to grow with `O(N)` in the number of tasks. This means that total running time grows `O(N^2)`.
I realize that as a workaround I can probably call `compute()` more frequently---based on the data below it looks like about every 10 steps would be optimal. But I still want to check if this is a known performance behavior of Dask and whether it's worth looking at the application more closely to see if I might be doing something wrong.
Thanks in advance and let me know if this would be better submitted under another Dask subproject.
| Steps | Tasks | Total Time (s) | Time per Task (ms) |
|-------|-------|----------------|--------------------|
| 1 | 32 | 0.5316095 | 16.61279688 |
| 10 | 320 | 0.6279482 | 1.962338125 |
| 50 | 1600 | 7.029356 | 4.3933475 |
| 100 | 3200 | 25.81444 | 8.0670125 |
| 200 | 6400 | 107.7474 | 16.83553125 |
| 300 | 9600 | 236.0527 | 24.58882292 |
```
$ conda list
dask 1.1.5 py_0
dask-core 1.1.5 py_0
distributed 1.26.1 py37_0
python 3.7.1 h0371630_7
tornado 6.0.2 py37h7b6447c_0
```
Contributor guide
Assessment
This issue has not been assessed yet.