Throttling tasks
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
To prevent too many tasks trying to use the same resources (e.g. database, web api) at the same time I would like to control the maximum degree of parallelism of certain jobs.
I understand that it should be possible to control this using queues as documented in [Data Streams with Queues](http://distributed.readthedocs.io/en/latest/queues.html#data-streams-with-queues) and as demonstrated in [this](http://stackoverflow.com/a/38921763/2960140) SO answer.
When I try a minimal example it doesn't seem if the `maxsize` argument is having any effect?
```python
from queue import Queue
import time
from distributed import Client
from distributed import as_completed
from numpy.random import rand
client = Client("10.200.13.45:8786")
def sleep(delay):
from time import sleep
sleep(delay)
return delay
delay = 5*rand(10)
in_q = Queue()
for t in delay:
in_q.put(t)
maxsize = 10
print("maxsize =", maxsize)
out_q = client.map(sleep, in_q, maxsize=maxsize)
t0 = time.time()
print()
futs = [out_q.get() for _ in range(len(delay))]
for fut in as_completed(futs):
print(fut.result(), flush=True)
t1 = time.time()
print()
print("Total Delay =", sum(delay))
print("Max Delay =", max(delay))
print("Time Taken =", t1 - t0)
```
When the `maxsize` is equal to the number of tasks they all run in parallel and the time taken is equal to (just over) the largest delay:
```
maxsize = 10
0.605000352854
0.87092424672
1.71378184661
1.91747345547
1.94673285822
1.98018309505
2.78900939666
3.42870653016
3.53337024226
4.46855449187
Total Delay = 23.2537365159
Max Delay = 4.46855449187
Time Taken = 4.508450746536255
```
When the `maxsize` is equal to 1 I would expect the tasks to be run sequentially and the time taken to equal the sum of all delays. It appears that the `maxsize` argument has no effect and that the tasks all run in parallel?
```
maxsize = 1
0.32444368678
0.996665256193
1.55266709866
1.86744982966
1.96963848636
2.51513007203
2.5978904159
3.15468744398
3.87090085426
4.28739030954
Total Delay = 23.1368634534
Max Delay = 4.28739030954
Time Taken = 4.312431335449219
```

Am I doing something really dumb or missing something obvious?
Contributor guide
Assessment
This issue has not been assessed yet.