dask / dask/distributed

`get_client` returns different Clients in different worker threads

Open
#5,466 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Similar to https://github.com/dask/distributed/issues/3827, `get_client` is not thread-safe—calling it within a task returns different Client instances in different threads.

```python
def test_get_client_threadsafe_sync():
def f(x):
return get_client().id

with cluster(nworkers=1, worker_kwargs={"nthreads": 4}) as (scheduler, workers):
with Client(scheduler["address"]) as client:
futures = client.map(f, range(100))
ids = client.gather(futures)
assert len(set(ids)) == 1
# > assert len(set(ids)) == 1
# E assert 4 == 1
# E +4
# E -1
```

If you are using a Client within a task, there's a good chance that Futures it produces will not be usable within another task. For example, if you create some Futures in one task, return them as the result of the task, and then try to reuse the Futures in a subsequent task, you may or may not find that you can't `get_client().gather(...)` those Futures.

This is a somewhat uncommon use-case, but if you are doing it, it's very difficult to debug.

FWIW doing
```python
client = get_worker().client
```
seems to avoid the issue.

**Environment**:

- Distributed version: a1b67b84226d3053517dffcb6f0c8fe8821eb8fb
- Python version: 3.9.5
- Operating System: macOS
- Install method (conda, pip, source): source

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.