using dask.array.full_like can raise strange "no Client active" error
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
Under dask and distributed 2023.6.0, using the `dask.array.full_like` function can lead to a strange error claiming that there is "no Client active." This occurs when a `LocalCluster` is used which has been created with more than one thread per worker. To reproduce this, use the following code:
```python
import dask.array as da
from distributed import Client, LocalCluster
def driver():
arr = da.random.default_rng().random((100, 100), chunks=(30, 30))
mean = da.nanmean(arr)
like_arr = da.full_like(arr, fill_value=mean).compute()
if __name__ == '__main__':
cluster = LocalCluster(n_workers=2, threads_per_worker=2)
client = Client(cluster)
driver()
```
Sometimes I have to run this multiple times to generate the failure, but it typically happens within 5 attempts. The error message looks like this:
```
File "/python/lib/python3.10/site-packages/dask/array/wrap.py", line 140, in _broadcast_trick_inner
return np.broadcast_to(func(meta, *args, shape=null_shape, **kwargs), shape)
File "<__array_function__ internals>", line 180, in full_like
File "/python/lib/python3.10/site-packages/numpy/core/numeric.py", line 423, in full_like
multiarray.copyto(res, fill_value, casting='unsafe')
File "<__array_function__ internals>", line 180, in copyto
File "/python/lib/python3.10/site-packages/dask/array/core.py", line 1750, in __array_function__
return handle_nonmatching_names(func, args, kwargs)
File "/python/lib/python3.10/site-packages/dask/array/core.py", line 1726, in handle_nonmatching_names
args, kwargs = compute(args, kwargs)
File "/python/lib/python3.10/site-packages/dask/base.py", line 583, in compute
schedule = get_scheduler(
File "/python/lib/python3.10/site-packages/dask/base.py", line 1403, in get_scheduler
return get_scheduler(scheduler=config.get("scheduler", None))
File "/python/lib/python3.10/site-packages/dask/base.py", line 1378, in get_scheduler
raise RuntimeError(
RuntimeError: Requested dask.distributed scheduler but no Client active.
```
I've truncated the stack trace somewhat but I can post a longer one that goes back to the original call if that would be useful.
This toy example is distilled from a real example that we have in our codebase. We have not observed this failure with non-local clusters, but it does crop up with some regularity in CI, where we use a `LocalCluster` fixture for our testing. This is the only place where this error happens, and it goes away if I replace `da.full_like(arr, fill_value=mean)` with `da.full(arr.shape, fill_value=mean)`. The error also goes away if I create the `LocalCluster` with `threads_per_worker=1`.
When debugging this issue, I drilled deep down into the `distributed/client.py` file where `default_client` is called (the error which gives rise to this `RuntimeError`) and I noticed that placing a delay after the line `c = c or _get_global_client()` eliminated this error; I discovered this by accident while trying to dump some information about the state of the client to a file.
**Environment**:
- Dask version: 2023.6.0
- Python version: 3.10
- Operating System: OS X, as well as linux/aarch64 (docker)
- Install method (conda, pip, source): pip
Contributor guide
Assessment
This issue has not been assessed yet.