Unique keys runs slow with time
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I have a DAG that I want to run in parallel multiple times, but each time with a different input params
Having the same key for all of the parallel runs isn't working, because it will only run it once with the first run params
My solution for that was to generate a guid as a key for each task
But this issue is that having guid is the key creates huge number of tasks, and the scheduler becomes slower with time
Trying to have a pool of keys solved the issue, but this isn’t the solution I want to have
Once the task finishes, I don't care about it anymore so if deleting the keys is an option and will solve the issue, it will be great
```python
import asyncio
from random import randint
from uuid import uuid4
import time
from dask.distributed import Client
def add(x, y):
return x + y
async def do_once(n):
def get_key(name):
# return name # not working, running one once
# return f'{name}_{uuid4().hex}' # become slow after few minutes
return f'{name}_{n}' # works good
async with Client(address='0.0.0.0:8786', asynchronous=True) as client:
n_key = get_key('n')
x_key = get_key('x')
y_key = get_key('y')
z_key = get_key('z')
ran = randint(0, 100)
dsk = {
n_key: n,
x_key: (add, n_key, 1),
y_key: (add, n_key, ran),
z_key: (add, x_key, y_key)
}
futures = client.get(dsk, keys=[z_key], sync=False)
res = await futures[0]
assert res == ((n + 1) + (n + ran)), f"{res} != {((n + 1) + (n + ran))}"
async def main():
while True:
start = time.time()
futures = [do_once(j) for j in range(20)]
await asyncio.gather(*futures)
print(f'iteration took {time.time() - start}')
asyncio.run(main())
```
**Environment**:
- Dask version: 2022.02.1
- Python version: 3.8.7
- Operating System: MacOS
- Install method (conda, pip, source): poetry
Contributor guide
Assessment
This issue has not been assessed yet.