dask / dask/distributed

Client returned by distributed.get_client() on LocalCluster worker has unknown properties

Open
#1,820 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

The client returned by distributed.get_client() on a LocalCluster worker appears to be the same client used by the original client which created the cluster, as opposed to a standard synchronous client. For example, if you create a local cluster using AioClient() or Client(asynchronous=True), it appears as if the result of distributed.get_client() within a synchronous function called in a task will be a reference to that object. This is a problem because, as you are now in a synchronous context in a running event loop, it becomes hard to write code which could be run on both a LocalCluster and a connection to a remote cluster. Is this intended behavior? As a workaround, I'm calling the following to ensure that the client is always the same, although one would expect this to have some overhead:

```python
from distributed import get_client, Client

def f():
existing = get_client() # This may be an AioClient, or a Client() or a Client(asynchronous=True)
client = Client(address=existing.scheduler.address)

# do things with client

client.close()
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.