xr.merge bug? when using combine_attrs='drop_conflicts'
@keewis is already working on this.
Since Jul 29, 2021.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What happened:
I have recently encountered a situation where combining two datasets failed, due to the datatype of their attributes.
This example illustrates the situation:
ds1 = xr.Dataset(attrs={'a':[5]})
ds2 = xr.Dataset(attrs={'a':6})
xr.merge([ds1, ds2], combine_attrs='drop_conflicts')
give me this error:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-12-1c8e82be0882> in <module>
2 ds2 = xr.Dataset(attrs={'a':6})
3
----> 4 xr.merge([ds1, ds2], combine_attrs='drop_conflicts')
/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/merge.py in merge(objects, compat, join, fill_value, combine_attrs)
898 dict_like_objects.append(obj)
899
--> 900 merge_result = merge_core(
901 dict_like_objects,
902 compat,
/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/merge.py in merge_core(objects, compat, join, combine_attrs, priority_arg, explicit_coords, indexes, fill_value)
654 )
655
--> 656 attrs = merge_attrs(
657 [var.attrs for var in coerced if isinstance(var, (Dataset, DataArray))],
658 combine_attrs,
/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/merge.py in merge_attrs(variable_attrs, combine_attrs, context)
544 }
545 )
--> 546 result = {
547 key: value
548 for key, value in result.items()
/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/merge.py in <dictcomp>(.0)
547 key: value
548 for key, value in result.items()
--> 549 if key not in attrs or equivalent(attrs[key], value)
550 }
551 dropped_keys |= {key for key in attrs if key not in result}
/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/utils.py in equivalent(first, second)
171 return duck_array_ops.array_equiv(first, second)
172 elif isinstance(first, list) or isinstance(second, list):
--> 173 return list_equiv(first, second)
174 else:
175 return (
/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/utils.py in list_equiv(first, second)
182 def list_equiv(first, second):
183 equiv = True
--> 184 if len(first) != len(second):
185 return False
186 else:
TypeError: object of type 'int' has no len()
Took me a while to find out what the root cause of this was with a fully populated dataset, since the error is less than obvious.
What you expected to happen:
In my understanding this should just drop the attribute a.
The example works like expected when both attributes are an integer or both are lists with an integer.
The error is only triggered when the type is mixed.
Is there a way to handle this case more elegantly?
Output of xr.show_versions()
INSTALLED VERSIONS ------------------ commit: None python: 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) [GCC 9.3.0] python-bits: 64 OS: Linux OS-release: 5.4.89+ machine: x86_64 processor: x86_64 byteorder: little LC_ALL: C.UTF-8 LANG: C.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.10.6 libnetcdf: 4.7.4xarray: 0.19.1.dev8+gda99a566
pandas: 1.2.4
numpy: 1.20.2
scipy: 1.6.2
netCDF4: 1.5.6
pydap: installed
h5netcdf: 0.11.0
h5py: 3.2.1
Nio: None
zarr: 2.7.1
cftime: 1.4.1
nc_time_axis: 1.2.0
PseudoNetCDF: None
rasterio: 1.2.2
cfgrib: 0.9.9.0
iris: None
bottleneck: 1.3.2
dask: 2021.04.1
distributed: 2021.04.1
matplotlib: 3.4.1
cartopy: 0.19.0
seaborn: None
numbagg: None
pint: 0.17
setuptools: 49.6.0.post20210108
pip: 20.3.4
conda: None
pytest: None
IPython: 7.22.0
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.