pydata / pydata/xarray

xr.merge bug? when using combine_attrs='drop_conflicts'

Open
#5,649 3 comments 1 reaction 1 assignee View on GitHub

@keewis is already working on this.

Since Jul 29, 2021.

bug topic-combine
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened:
I have recently encountered a situation where combining two datasets failed, due to the datatype of their attributes.
This example illustrates the situation:

ds1 = xr.Dataset(attrs={'a':[5]})
ds2 = xr.Dataset(attrs={'a':6})

xr.merge([ds1, ds2], combine_attrs='drop_conflicts')

give me this error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-12-1c8e82be0882> in <module>
      2 ds2 = xr.Dataset(attrs={'a':6})
      3 
----> 4 xr.merge([ds1, ds2], combine_attrs='drop_conflicts')

/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/merge.py in merge(objects, compat, join, fill_value, combine_attrs)
    898         dict_like_objects.append(obj)
    899 
--> 900     merge_result = merge_core(
    901         dict_like_objects,
    902         compat,

/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/merge.py in merge_core(objects, compat, join, combine_attrs, priority_arg, explicit_coords, indexes, fill_value)
    654         )
    655 
--> 656     attrs = merge_attrs(
    657         [var.attrs for var in coerced if isinstance(var, (Dataset, DataArray))],
    658         combine_attrs,

/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/merge.py in merge_attrs(variable_attrs, combine_attrs, context)
    544                 }
    545             )
--> 546             result = {
    547                 key: value
    548                 for key, value in result.items()

/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/merge.py in <dictcomp>(.0)
    547                 key: value
    548                 for key, value in result.items()
--> 549                 if key not in attrs or equivalent(attrs[key], value)
    550             }
    551             dropped_keys |= {key for key in attrs if key not in result}

/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/utils.py in equivalent(first, second)
    171         return duck_array_ops.array_equiv(first, second)
    172     elif isinstance(first, list) or isinstance(second, list):
--> 173         return list_equiv(first, second)
    174     else:
    175         return (

/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/utils.py in list_equiv(first, second)
    182 def list_equiv(first, second):
    183     equiv = True
--> 184     if len(first) != len(second):
    185         return False
    186     else:

TypeError: object of type 'int' has no len()

Took me a while to find out what the root cause of this was with a fully populated dataset, since the error is less than obvious.

What you expected to happen:
In my understanding this should just drop the attribute a.
The example works like expected when both attributes are an integer or both are lists with an integer.
The error is only triggered when the type is mixed.

Is there a way to handle this case more elegantly?

Output of xr.show_versions() INSTALLED VERSIONS ------------------ commit: None python: 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) [GCC 9.3.0] python-bits: 64 OS: Linux OS-release: 5.4.89+ machine: x86_64 processor: x86_64 byteorder: little LC_ALL: C.UTF-8 LANG: C.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.10.6 libnetcdf: 4.7.4

xarray: 0.19.1.dev8+gda99a566
pandas: 1.2.4
numpy: 1.20.2
scipy: 1.6.2
netCDF4: 1.5.6
pydap: installed
h5netcdf: 0.11.0
h5py: 3.2.1
Nio: None
zarr: 2.7.1
cftime: 1.4.1
nc_time_axis: 1.2.0
PseudoNetCDF: None
rasterio: 1.2.2
cfgrib: 0.9.9.0
iris: None
bottleneck: 1.3.2
dask: 2021.04.1
distributed: 2021.04.1
matplotlib: 3.4.1
cartopy: 0.19.0
seaborn: None
numbagg: None
pint: 0.17
setuptools: 49.6.0.post20210108
pip: 20.3.4
conda: None
pytest: None
IPython: 7.22.0
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.