dask / dask/distributed

No clients found when multiple local clusters started

Open
#2,044 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

On 1.21.8

Error is:

```
Traceback (most recent call last):
File "testme4.py", line 65, in
test()
File "testme4.py", line 56, in test
do_gather(c)
File "testme4.py", line 36, in do_gather
wait(futs)
File "distributed/distributed/client.py", line 3337, in wait
client = default_client()
File "distributed/distributed/client.py", line 3581, in default_client
raise ValueError("No clients found\n"
ValueError: No clients found
Start an client and point it to the scheduler address
from distributed import Client
client = Client('ip-addr-of-scheduler:8786')
```

Repro is:

```
from distributed import LocalCluster, Client, worker_client, wait

import numpy as np
import pandas as pd
import time

def task2(nrows):
df = pd.DataFrame(np.ones([nrows, 100]))
return df

def task1():
with worker_client() as wc:
futs = []

for sz in range(1, 4):
f1 = wc.submit(task2, sz)
futs.append(f1)

wait(futs)

y = wc.gather(futs)

return pd.concat(y)

def do_gather(c: Client):
futs = []

for i in range(10):
futs.append(c.submit(task1))

wait(futs)

y = c.gather(futs)

return y

def test():
lc = LocalCluster(n_workers=2, silence_logs=True, memory_limit=1 << 30)
c = Client(lc)

do_gather(c)

lc2 = LocalCluster(n_workers=2, silence_logs=True, memory_limit=1 << 30)
c2 = Client(lc2)

do_gather(c2)

c2.close()
lc2.close()

do_gather(c)

c.close()
lc.close()

print("Done.")

if __name__ == '__main__':
test()
```

It would be nice to pass the correct client to `wait` so this will work.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.