`except Exception` in `compute_meta` makes custom functions and schedulers difficult
- Dominant language
- Python
- Stars
- 13.9k
- Forks
- 2k
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the issue**:
There is an `except Exception` block in `compute_meta` when running the function being called.
https://github.com/dask/dask/blame/1203b1bb6d52b1fb00d54656af4af8e6e35aa615/dask/array/utils.py#L162-L163
The commit message for this change mentions avoiding numpy warnings, but those should already be ignored by the `errstate` at the top of the function.
https://github.com/dask/dask/commit/aa5d4ac8eefa4cbcadd0ca112651d53b1403e0a8
In Satpy we use a `CustomScheduler` class in our testing and debugging to verify that we are not computing dask arrays until we expect to. We do this by raising a `RuntimeError` when the scheduler is told to compute more than N times. See:
https://github.com/pytroll/satpy/blob/bc32c9434815365d180b1c6d38c00b2b6b4d5f7e/satpy/tests/utils.py#L275-L290
However, it was recently discovered that this doesn't work if any of our functions require computing the meta array (ex. map_blocks, etc). Most recently it was when passing `xarray.DataArray`s to a `da.where` function by mistake (we're still not sure if this is a problem).
**Minimal Complete Verifiable Example**:
```python
import dask
import dask.array as da
import xarray as xr
class CustomScheduler(object):
"""Scheduler raising an exception if data are computed too many times."""
def __init__(self, max_computes=1):
"""Set starting and maximum compute counts."""
self.max_computes = max_computes
self.total_computes = 0
def __call__(self, dsk, keys, **kwargs):
"""Compute dask task and keep track of number of times we do so."""
import dask
self.total_computes += 1
if self.total_computes > self.max_computes:
raise RuntimeError("Too many dask computations were scheduled: "
"{}".format(self.total_computes))
return dask.get(dsk, keys, **kwargs)
with dask.config.set(scheduler=CustomScheduler(1)):
da.where(xr.DataArray(da.zeros((5, 5))), xr.DataArray(da.zeros((5, 5))), xr.DataArray(da.ones((5, 5)))).compute()
```
Exception:
```
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
Cell In[22], line 2
1 with dask.config.set(scheduler=CustomScheduler(1)):
----> 2 da.where(xr.DataArray(da.zeros((5, 5))), xr.DataArray(da.zeros((5, 5))), xr.DataArray(da.ones((5, 5)))).compute()
File ~/miniconda3/envs/polar2grid_py310/lib/python3.10/site-packages/dask/base.py:312, in DaskMethodsMixin.compute(self, **kwargs)
288 def compute(self, **kwargs):
289 """Compute this dask collection
290
291 This turns a lazy Dask collection into its in-memory equivalent.
(...)
310 dask.base.compute
311 """
--> 312 (result,) = compute(self, traverse=False, **kwargs)
313 return result
File ~/miniconda3/envs/polar2grid_py310/lib/python3.10/site-packages/dask/base.py:600, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
597 keys.append(x.__dask_keys__())
598 postcomputes.append(x.__dask_postcompute__())
--> 600 results = schedule(dsk, keys, **kwargs)
601 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
Cell In[13], line 14, in CustomScheduler.__call__(self, dsk, keys, **kwargs)
12 self.total_computes += 1
13 if self.total_computes > self.max_computes:
---> 14 raise RuntimeError("Too many dask computations were scheduled: "
15 "{}".format(self.total_computes))
16 return dask.get(dsk, keys, **kwargs)
RuntimeError: Too many dask computations were scheduled: 4
```
Note how the error message says there were 4 computations when there should have only been 2 when the `if self.total_computes > self.max_computes` check is triggered. This is due to `compute_meta` computing the DataArrays as far as I can tell and hiding the `RuntimeError` from trickling up to the user.
**Anything else we need to know?**:
**Environment**:
- Dask version: 2023.9.2 (above code run with a 2022.05.2 but 2023.9.2 also produces it)
- Python version: 3.10 and 3.11
- Operating System: Linux
- Install method (conda, pip, source): conda-forge
Contributor guide
Research direction
Start with dask/array/utils.py at compute_meta and run the copy-pastable CustomScheduler example from the issue. Trace how the function call and scheduler exception are handled, then check that the reported RuntimeError is no longer hidden while the existing NumPy warning behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100