dask / dask/distributed

Unique keys runs slow with time

Open
#5,965 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I have a DAG that I want to run in parallel multiple times, but each time with a different input params
Having the same key for all of the parallel runs isn't working, because it will only run it once with the first run params
My solution for that was to generate a guid as a key for each task
But this issue is that having guid is the key creates huge number of tasks, and the scheduler becomes slower with time
Trying to have a pool of keys solved the issue, but this isn’t the solution I want to have

Once the task finishes, I don't care about it anymore so if deleting the keys is an option and will solve the issue, it will be great

```python
import asyncio
from random import randint
from uuid import uuid4
import time
from dask.distributed import Client

def add(x, y):
return x + y

async def do_once(n):
def get_key(name):
# return name # not working, running one once
# return f'{name}_{uuid4().hex}' # become slow after few minutes
return f'{name}_{n}' # works good

async with Client(address='0.0.0.0:8786', asynchronous=True) as client:
n_key = get_key('n')
x_key = get_key('x')
y_key = get_key('y')
z_key = get_key('z')
ran = randint(0, 100)

dsk = {
n_key: n,
x_key: (add, n_key, 1),
y_key: (add, n_key, ran),
z_key: (add, x_key, y_key)
}

futures = client.get(dsk, keys=[z_key], sync=False)
res = await futures[0]

assert res == ((n + 1) + (n + ran)), f"{res} != {((n + 1) + (n + ran))}"

async def main():
while True:
start = time.time()
futures = [do_once(j) for j in range(20)]
await asyncio.gather(*futures)

print(f'iteration took {time.time() - start}')

asyncio.run(main())
```

**Environment**:

- Dask version: 2022.02.1
- Python version: 3.8.7
- Operating System: MacOS
- Install method (conda, pip, source): poetry

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.