pydata / pydata/xarray

Error concatenating Multiindex variables

Open
#3,659 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-combine
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

MCVE Code Sample
>>> import xarray as xr
>>> da = xr.DataArray([0, 1], dims=["location"], coords={"lat": ("location", [10, 11]), "lon": ("location", [20, 21])}).set_index(location=["lat", "lon"])
>>> da2 = xr.DataArray([2, 3], dims=["location"], coords={"lat": ("location", [12, 13]), "lon": ("location", [22, 23])}).set_index(location=["lat", "lon"])
>>> xr.concat([da["location"], da2["location"]], dim="location")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/harry/code/xarray/xarray/core/concat.py", line 135, in concat
    return f(objs, dim, data_vars, coords, compat, positions, fill_value, join)
  File "/home/harry/code/xarray/xarray/core/concat.py", line 431, in _dataarray_concat
    ds = _dataset_concat(
  File "/home/harry/code/xarray/xarray/core/concat.py", line 384, in _dataset_concat
    result = Dataset(result_vars, attrs=result_attrs)
  File "/home/harry/code/xarray/xarray/core/dataset.py", line 541, in __init__
    variables, coord_names, dims, indexes = merge_data_and_coords(
  File "/home/harry/code/xarray/xarray/core/merge.py", line 466, in merge_data_and_coords
    return merge_core(
  File "/home/harry/code/xarray/xarray/core/merge.py", line 556, in merge_core
    assert_unique_multiindex_level_names(variables)
  File "/home/harry/code/xarray/xarray/core/variable.py", line 2363, in assert_unique_multiindex_level_names
    raise ValueError("conflicting MultiIndex level name(s):\n%s" % conflict_str)
ValueError: conflicting MultiIndex level name(s):
'lat' (location), 'lat' (<this-array>)
'lon' (location), 'lon' (<this-array>)
Expected Output

The output should be the same as first concatenating the DataArrays, then extracting the dimension location:

>>> xr.concat([da, da2], dim="location")["location"]
<xarray.DataArray 'location' (location: 4)>
array([(10, 20), (11, 21), (12, 22), (13, 23)], dtype=object)
Coordinates:
  * location  (location) MultiIndex
  - lat       (location) int64 10 11 12 13
  - lon       (location) int64 20 21 22 23
Problem Description
>>> # da["location"] looks like a normal DataArray
>>> location = da["location"]
>>> location
<xarray.DataArray 'location' (location: 2)>
array([(10, 20), (11, 21)], dtype=object)
Coordinates:
  * location  (location) MultiIndex
  - lat       (location) int64 10 11
  - lon       (location) int64 20 21
>>> # but in actual fact, the variable._data is a MultiIndex
>>> location.variable._data
PandasIndexAdapter(array=MultiIndex([(10, 20),
            (11, 21)],
           names=['lat', 'lon']), dtype=dtype('O'))

This is why an error is thrown: variable.assert_unique_multiindex_level_names gets passed two variables: location.variable (the DataArray data values), and also location["location"].variable (the coordinate values), which are both MultiIndexes.

Output of xr.show_versions()
INSTALLED VERSIONS ------------------ commit: b3d3b4480b7fb63402eb6c02103bb8d6c7dbf93a python: 3.8.0 | packaged by conda-forge | (default, Nov 22 2019, 19:11:38) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.4.0-18362-Microsoft machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: None libnetcdf: None

xarray: 0.14.1+36.gb3d3b44
pandas: 0.25.3
numpy: 1.18.0
scipy: None
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.9.1
distributed: 2.9.1
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
setuptools: 42.0.2.post20191201
pip: 19.3.1
conda: None
pytest: 5.3.2
IPython: None
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the MCVE with the concatenation path in xarray/core/concat.py, then inspect the merge and MultiIndex checks in xarray/core/merge.py and xarray/core/variable.py. Add or update a regression test for concatenating the extracted location DataArrays; done means it produces the four expected MultiIndex values without the conflicting-level-name error.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.