dask / dask/dask

using dask.array.full_like can raise strange "no Client active" error

Open
#10,452 4 comments 0 reactions 0 assignees View on GitHub
array needs attention needs info
Dominant language
Python
Stars
13.9k
Forks
2k
PR merge metrics
No merged PRs in 30d

Description

Under dask and distributed 2023.6.0, using the `dask.array.full_like` function can lead to a strange error claiming that there is "no Client active." This occurs when a `LocalCluster` is used which has been created with more than one thread per worker. To reproduce this, use the following code:

```python
import dask.array as da
from distributed import Client, LocalCluster

def driver():
arr = da.random.default_rng().random((100, 100), chunks=(30, 30))
mean = da.nanmean(arr)
like_arr = da.full_like(arr, fill_value=mean).compute()

if __name__ == '__main__':
cluster = LocalCluster(n_workers=2, threads_per_worker=2)
client = Client(cluster)

driver()
```

Sometimes I have to run this multiple times to generate the failure, but it typically happens within 5 attempts. The error message looks like this:

```
File "/python/lib/python3.10/site-packages/dask/array/wrap.py", line 140, in _broadcast_trick_inner
return np.broadcast_to(func(meta, *args, shape=null_shape, **kwargs), shape)
File "<__array_function__ internals>", line 180, in full_like
File "/python/lib/python3.10/site-packages/numpy/core/numeric.py", line 423, in full_like
multiarray.copyto(res, fill_value, casting='unsafe')
File "<__array_function__ internals>", line 180, in copyto
File "/python/lib/python3.10/site-packages/dask/array/core.py", line 1750, in __array_function__
return handle_nonmatching_names(func, args, kwargs)
File "/python/lib/python3.10/site-packages/dask/array/core.py", line 1726, in handle_nonmatching_names
args, kwargs = compute(args, kwargs)
File "/python/lib/python3.10/site-packages/dask/base.py", line 583, in compute
schedule = get_scheduler(
File "/python/lib/python3.10/site-packages/dask/base.py", line 1403, in get_scheduler
return get_scheduler(scheduler=config.get("scheduler", None))
File "/python/lib/python3.10/site-packages/dask/base.py", line 1378, in get_scheduler
raise RuntimeError(
RuntimeError: Requested dask.distributed scheduler but no Client active.
```

I've truncated the stack trace somewhat but I can post a longer one that goes back to the original call if that would be useful.

This toy example is distilled from a real example that we have in our codebase. We have not observed this failure with non-local clusters, but it does crop up with some regularity in CI, where we use a `LocalCluster` fixture for our testing. This is the only place where this error happens, and it goes away if I replace `da.full_like(arr, fill_value=mean)` with `da.full(arr.shape, fill_value=mean)`. The error also goes away if I create the `LocalCluster` with `threads_per_worker=1`.

When debugging this issue, I drilled deep down into the `distributed/client.py` file where `default_client` is called (the error which gives rise to this `RuntimeError`) and I noticed that placing a delay after the line `c = c or _get_global_client()` eliminated this error; I discovered this by accident while trying to dump some information about the state of the client to a file.

**Environment**:

- Dask version: 2023.6.0
- Python version: 3.10
- Operating System: OS X, as well as linux/aarch64 (docker)
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.