Chunking causes unrelated non-dimension coordinate to become a dask array
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened:
Rechunking along an independent dimension causes unrelated non-dimension coordinates to become dask arrays. The dimension coordinates do not seem affected.
I can stick in a synchronous compute on the coordinate to recover, but wanted to be sure this was the expected behavior.
What you expected to happen:
Chunking along an unrelated dimension should not affect unrelated non-dimension coordinates.
Minimal Complete Verifiable Example:
import xarray as xr
import dask.array as da
def print_coords(a, title):
print()
print(title)
for dim in ['x', 'y', 'b']:
if dim in a.dims or dim in a.coords:
print('dim:', dim, 'type:', type(a.coords[dim].data))
arr = xr.DataArray(da.zeros((20, 20), chunks=10), dims=('x', 'y'),
coords={'b': ('y', range(100,120)),
'x': range(20),
'y': range(20)})
print_coords(arr, 'Original')
# The following line rechunks independently of b or y.
# Removing this line allows the code to succeed.
arr = arr.chunk({'x': 5})
print_coords(arr, 'After chunking')
arr = arr.sel(y=2)
print_coords(arr, 'After selection')
print()
print('Scalar values:')
print('y=', arr.coords['y'].item())
print('b=', arr.coords['b'].item()) # Sad Panda
Original
dim: x type: <class 'numpy.ndarray'>
dim: y type: <class 'numpy.ndarray'>
dim: b type: <class 'numpy.ndarray'>
After chunking
dim: x type: <class 'numpy.ndarray'>
dim: y type: <class 'numpy.ndarray'>
dim: b type: <class 'dask.array.core.Array'>
After selection
dim: x type: <class 'numpy.ndarray'>
dim: y type: <class 'numpy.ndarray'>
dim: b type: <class 'dask.array.core.Array'>
Scalar values:
y= 2
<stack trace elided>
NotImplementedError: 'item' is not yet a valid method on dask arrays
Environment:
Output of xr.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.7.6 | packaged by conda-forge | (default, Jun 1 2020, 18:57:50)
[GCC 7.5.0]
python-bits: 64
OS: Linux
OS-release: 4.19.112+
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.4
libnetcdf: None
xarray: 0.15.1
pandas: 1.0.5
numpy: 1.18.5
scipy: 1.4.1
netCDF4: None
pydap: None
h5netcdf: None
h5py: 2.10.0
Nio: None
zarr: 2.4.0
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.19.0
distributed: 2.19.0
matplotlib: 3.2.2
cartopy: None
seaborn: None
numbagg: None
setuptools: 49.1.0.post20200704
pip: 20.1.1
conda: 4.8.3
pytest: 5.4.3
IPython: 7.16.1
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Run the minimal complete verifiable example with DataArray.chunk({'x': 5}) and the subsequent sel(y=2) call to reproduce the dask-backed b coordinate and failing item() access. Trace the chunking and selection entry points, then verify that rechunking an unrelated dimension leaves unrelated non-dimension coordinates usable as before.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100