dask / dask/distributed

Throttling tasks

Open
#864 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

To prevent too many tasks trying to use the same resources (e.g. database, web api) at the same time I would like to control the maximum degree of parallelism of certain jobs.

I understand that it should be possible to control this using queues as documented in [Data Streams with Queues](http://distributed.readthedocs.io/en/latest/queues.html#data-streams-with-queues) and as demonstrated in [this](http://stackoverflow.com/a/38921763/2960140) SO answer.

When I try a minimal example it doesn't seem if the `maxsize` argument is having any effect?
```python
from queue import Queue
import time

from distributed import Client
from distributed import as_completed
from numpy.random import rand

client = Client("10.200.13.45:8786")

def sleep(delay):
from time import sleep
sleep(delay)
return delay

delay = 5*rand(10)

in_q = Queue()
for t in delay:
in_q.put(t)

maxsize = 10
print("maxsize =", maxsize)
out_q = client.map(sleep, in_q, maxsize=maxsize)

t0 = time.time()
print()
futs = [out_q.get() for _ in range(len(delay))]
for fut in as_completed(futs):
print(fut.result(), flush=True)
t1 = time.time()

print()
print("Total Delay =", sum(delay))
print("Max Delay =", max(delay))
print("Time Taken =", t1 - t0)
```
When the `maxsize` is equal to the number of tasks they all run in parallel and the time taken is equal to (just over) the largest delay:
```
maxsize = 10

0.605000352854
0.87092424672
1.71378184661
1.91747345547
1.94673285822
1.98018309505
2.78900939666
3.42870653016
3.53337024226
4.46855449187

Total Delay = 23.2537365159
Max Delay = 4.46855449187
Time Taken = 4.508450746536255
```
When the `maxsize` is equal to 1 I would expect the tasks to be run sequentially and the time taken to equal the sum of all delays. It appears that the `maxsize` argument has no effect and that the tasks all run in parallel?
```
maxsize = 1

0.32444368678
0.996665256193
1.55266709866
1.86744982966
1.96963848636
2.51513007203
2.5978904159
3.15468744398
3.87090085426
4.28739030954

Total Delay = 23.1368634534
Max Delay = 4.28739030954
Time Taken = 4.312431335449219
```
![image](https://cloud.githubusercontent.com/assets/881019/22780070/095a451c-ef09-11e6-88e7-a66d4db9b3f6.png)

Am I doing something really dumb or missing something obvious?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.