map_blocks raises AssertionError given chunks along a multiindex
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened?
To parallelize a computation over a sparse array, I want to create chunks after stacking the array and dropping fill values. Such an array has a multiindex, which breaks the map_blocks method (with an unhelpful error). The AssertionError raised suggests to me that the method does not account for such an array having (coordinate) variables that are not dimensions or a scalar.
What did you expect to happen?
I expect the MCVE below to return its input unmodified.
Minimal Complete Verifiable Example
import xarray as xr
a = xr.DataArray([[0, 1], [2, 3]], {"x": [0, 1], "y": [0, 1]})
a = a.stack({"n": ("x", "y")})
a = a.chunk({"n": 2})
a.map_blocks(lambda x: x)
MVCE confirmation
- Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- Complete example — the example is self-contained, including all data and the text of any traceback.
- Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
- New issue — a search of GitHub Issues suggests this is not a duplicate.
- Recent environment — the issue occurs with the latest version of xarray and its dependencies.
Relevant log output
File ~/tmp/bug/venv/lib/python3.11/site-packages/xarray/core/dataarray.py:5526, in DataArray.map_blocks(self, func, args, kwargs, template)
5428 """
5429 Apply a function to each block of this DataArray.
5430
(...)
5522 month (time) int64 dask.array<chunksize=(24,), meta=np.ndarray>
5523 """
5524 from xarray.core.parallel import map_blocks
-> 5526 return map_blocks(func, self, args, kwargs, template)
File ~/tmp/bug/venv/lib/python3.11/site-packages/xarray/core/parallel.py:539, in map_blocks(func, obj, args, kwargs, template)
535 for chunk_tuple in itertools.product(*ichunk.values()):
536 # mapping from dimension name to chunk index
537 chunk_index = dict(zip(ichunk.keys(), chunk_tuple))
--> 539 blocked_args = [
540 subset_dataset_to_block(graph, gname, arg, input_chunk_bounds, chunk_index)
541 if isxr
542 else arg
543 for isxr, arg in zip(is_xarray, npargs)
544 ]
546 # raise nice error messages in _wrapper
547 expected: ExpectedDict = {
548 # input chunk 0 along a dimension maps to output chunk 0 along the same dimension
549 # even if length of dimension is changed by the applied function
(...)
563 },
564 }
File ~/tmp/bug/venv/lib/python3.11/site-packages/xarray/core/parallel.py:540, in <listcomp>(.0)
535 for chunk_tuple in itertools.product(*ichunk.values()):
536 # mapping from dimension name to chunk index
537 chunk_index = dict(zip(ichunk.keys(), chunk_tuple))
539 blocked_args = [
--> 540 subset_dataset_to_block(graph, gname, arg, input_chunk_bounds, chunk_index)
541 if isxr
542 else arg
543 for isxr, arg in zip(is_xarray, npargs)
544 ]
546 # raise nice error messages in _wrapper
547 expected: ExpectedDict = {
548 # input chunk 0 along a dimension maps to output chunk 0 along the same dimension
549 # even if length of dimension is changed by the applied function
(...)
563 },
564 }
File ~/tmp/bug/venv/lib/python3.11/site-packages/xarray/core/parallel.py:195, in subset_dataset_to_block(graph, gname, dataset, input_chunk_bounds, chunk_index)
190 graph[chunk_variable_task] = (
191 tuple,
192 [variable.dims, chunk, variable.attrs],
193 )
194 else:
--> 195 assert name in dataset.dims or variable.ndim == 0
197 # non-dask array possibly with dimensions chunked on other variables
198 # index into variable appropriately
199 subsetter = {
200 dim: _get_chunk_slicer(dim, chunk_index, input_chunk_bounds)
201 for dim in variable.dims
202 }
AssertionError:
Anything else we need to know?
No response
Environment
INSTALLED VERSIONS
commit: None
python: 3.11.6 (main, Nov 2 2023, 04:52:24) [Clang 14.0.3 (clang-1403.0.22.14.1)]
python-bits: 64
OS: Darwin
OS-release: 22.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.12.2
libnetcdf: 4.9.3-development
xarray: 2024.1.1
pandas: 2.2.0
numpy: 1.26.4
scipy: 1.11.4
netCDF4: 1.6.5
pydap: None
h5netcdf: 1.3.0
h5py: 3.10.0
Nio: None
zarr: 2.16.1
cftime: 1.6.3
nc_time_axis: None
iris: None
bottleneck: None
dask: 2024.2.0
distributed: None
matplotlib: 3.8.2
cartopy: 0.22.0
seaborn: None
numbagg: None
fsspec: 2024.2.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 68.2.2
pip: 24.0
conda: None
pytest: None
mypy: None
IPython: 8.18.1
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in xarray/core/parallel.py at subset_dataset_to_block, where the AssertionError occurs for the stacked DataArray's coordinate variables. Reproduce the issue with the provided map_blocks example and trace how the multiindex variables are handled. Done means the example returns its input unmodified without the assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100