No clients found when multiple local clusters started
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
On 1.21.8
Error is:
```
Traceback (most recent call last):
File "testme4.py", line 65, in
test()
File "testme4.py", line 56, in test
do_gather(c)
File "testme4.py", line 36, in do_gather
wait(futs)
File "distributed/distributed/client.py", line 3337, in wait
client = default_client()
File "distributed/distributed/client.py", line 3581, in default_client
raise ValueError("No clients found\n"
ValueError: No clients found
Start an client and point it to the scheduler address
from distributed import Client
client = Client('ip-addr-of-scheduler:8786')
```
Repro is:
```
from distributed import LocalCluster, Client, worker_client, wait
import numpy as np
import pandas as pd
import time
def task2(nrows):
df = pd.DataFrame(np.ones([nrows, 100]))
return df
def task1():
with worker_client() as wc:
futs = []
for sz in range(1, 4):
f1 = wc.submit(task2, sz)
futs.append(f1)
wait(futs)
y = wc.gather(futs)
return pd.concat(y)
def do_gather(c: Client):
futs = []
for i in range(10):
futs.append(c.submit(task1))
wait(futs)
y = c.gather(futs)
return y
def test():
lc = LocalCluster(n_workers=2, silence_logs=True, memory_limit=1 << 30)
c = Client(lc)
do_gather(c)
lc2 = LocalCluster(n_workers=2, silence_logs=True, memory_limit=1 << 30)
c2 = Client(lc2)
do_gather(c2)
c2.close()
lc2.close()
do_gather(c)
c.close()
lc.close()
print("Done.")
if __name__ == '__main__':
test()
```
It would be nice to pass the correct client to `wait` so this will work.
Contributor guide
Assessment
This issue has not been assessed yet.